Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 1476762 Details for
Bug 1618932
dht-selfheal.c: Directory selfheal failed
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
gfapi log
gfapi_20180818_unlock_error.log (text/plain), 2.53 MB, created by
frostyplanet
on 2018-08-18 12:49:41 UTC
(
hide
)
Description:
gfapi log
Filename:
MIME Type:
Creator:
frostyplanet
Created:
2018-08-18 12:49:41 UTC
Size:
2.53 MB
patch
obsolete
>[2018-08-18 11:35:01.133362] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:01.133412] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:01.133474] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t at vol0-replicate-24 >[2018-08-18 11:35:01.133920] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:01.134002] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t returned with op_ret 0 >[2018-08-18 11:35:01.134645] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7fc073b634e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7fc082b88631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7fc07374e820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:01.134706] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 11:35:01.134680] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:01.134705] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: 6135251d-bd90-46da-a007-650da6fbe9d0: generation now vs cached: 3, 3 >[2018-08-18 11:35:01.134980] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:01.135816] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:01.135869] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:01.135895] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:01.136198] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:01.136974] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:01.137211] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:01.137417] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:01.138871] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:01.138909] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:01.138932] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:01.139031] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:01.139066] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:01.140539] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:01.140589] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:01.140634] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:01.141161] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:01.141363] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 11:35:01.141390] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:01.141768] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:01.142007] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:01.142319] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:01.142347] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:01.142409] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): picking the request for winding >[2018-08-18 11:35:01.142485] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:01.142506] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:01.142779] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:01.143020] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:01.143225] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:01.143268] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): picking the request for winding >[2018-08-18 11:35:01.143290] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:01.143351] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:01.143369] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:01.143420] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:01.143809] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:01.143895] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:01.144086] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:01.144130] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): picking the request for winding >[2018-08-18 11:35:01.144150] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:01.144211] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:01.144227] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:01.144286] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:01.144665] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:01.144869] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:01.145241] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:01.145254] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:01.145271] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:01.145610] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:01.145647] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:03.186055] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on vol0-replicate-7 >[2018-08-18 11:35:03.186536] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49195: ping timer event already removed >[2018-08-18 11:35:03.186848] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49195: ping timer event already removed >[2018-08-18 11:35:03.187105] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49195: ping timer event already removed >[2018-08-18 11:35:03.187199] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m with op_ret 0 >[2018-08-18 11:35:03.187521] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49188: ping timer event already removed >[2018-08-18 11:35:03.187576] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.187729] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49188: ping timer event already removed >[2018-08-18 11:35:03.187784] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.187922] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49188: ping timer event already removed >[2018-08-18 11:35:03.187984] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.188023] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.188038] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on vol0-replicate-0 returned error [No such file or directory] >[2018-08-18 11:35:03.188112] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49189: ping timer event already removed >[2018-08-18 11:35:03.188217] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.188254] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49189: ping timer event already removed >[2018-08-18 11:35:03.188336] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.188396] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49189: ping timer event already removed >[2018-08-18 11:35:03.188468] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.188492] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.188503] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on vol0-replicate-1 returned error [No such file or directory] >[2018-08-18 11:35:03.188568] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49190: ping timer event already removed >[2018-08-18 11:35:03.188667] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.188712] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49190: ping timer event already removed >[2018-08-18 11:35:03.188789] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.188850] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49190: ping timer event already removed >[2018-08-18 11:35:03.188923] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.188953] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.188965] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on vol0-replicate-2 returned error [No such file or directory] >[2018-08-18 11:35:03.189010] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49191: ping timer event already removed >[2018-08-18 11:35:03.189106] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.189158] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49191: ping timer event already removed >[2018-08-18 11:35:03.189231] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.189337] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49191: ping timer event already removed >[2018-08-18 11:35:03.189439] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.189485] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.189503] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49192: ping timer event already removed >[2018-08-18 11:35:03.189507] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on vol0-replicate-3 returned error [No such file or directory] >[2018-08-18 11:35:03.189628] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.189654] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49192: ping timer event already removed >[2018-08-18 11:35:03.189724] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.189794] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49192: ping timer event already removed >[2018-08-18 11:35:03.189870] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.189892] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.189904] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on vol0-replicate-4 returned error [No such file or directory] >[2018-08-18 11:35:03.189953] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49193: ping timer event already removed >[2018-08-18 11:35:03.190019] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.190101] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49193: ping timer event already removed >[2018-08-18 11:35:03.190183] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.190235] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49193: ping timer event already removed >[2018-08-18 11:35:03.190425] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49194: ping timer event already removed >[2018-08-18 11:35:03.190476] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.190604] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49194: ping timer event already removed >[2018-08-18 11:35:03.190661] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.190789] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49194: ping timer event already removed >[2018-08-18 11:35:03.190836] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.190857] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.190870] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on vol0-replicate-6 returned error [No such file or directory] >[2018-08-18 11:35:03.191060] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49196: ping timer event already removed >[2018-08-18 11:35:03.191127] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.191265] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49196: ping timer event already removed >[2018-08-18 11:35:03.191314] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.191416] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49196: ping timer event already removed >[2018-08-18 11:35:03.191471] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.191486] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.191497] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on vol0-replicate-8 returned error [No such file or directory] >[2018-08-18 11:35:03.191614] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49197: ping timer event already removed >[2018-08-18 11:35:03.191665] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.191768] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49197: ping timer event already removed >[2018-08-18 11:35:03.191814] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.191918] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49197: ping timer event already removed >[2018-08-18 11:35:03.191972] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.191996] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.192005] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on vol0-replicate-9 returned error [No such file or directory] >[2018-08-18 11:35:03.192083] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49198: ping timer event already removed >[2018-08-18 11:35:03.192154] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.192225] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49198: ping timer event already removed >[2018-08-18 11:35:03.192290] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.192367] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49198: ping timer event already removed >[2018-08-18 11:35:03.192456] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.192479] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.192491] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on vol0-replicate-10 returned error [No such file or directory] >[2018-08-18 11:35:03.192565] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49199: ping timer event already removed >[2018-08-18 11:35:03.192616] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.192734] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49199: ping timer event already removed >[2018-08-18 11:35:03.192784] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.192886] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49199: ping timer event already removed >[2018-08-18 11:35:03.192932] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.192953] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.192961] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on vol0-replicate-11 returned error [No such file or directory] >[2018-08-18 11:35:03.193052] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49200: ping timer event already removed >[2018-08-18 11:35:03.193128] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.193199] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49200: ping timer event already removed >[2018-08-18 11:35:03.193273] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.193343] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49200: ping timer event already removed >[2018-08-18 11:35:03.193399] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.193440] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.193460] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on vol0-replicate-12 returned error [No such file or directory] >[2018-08-18 11:35:03.193532] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49201: ping timer event already removed >[2018-08-18 11:35:03.193557] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c013160, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.193581] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c013160, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.193757] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49201: ping timer event already removed >[2018-08-18 11:35:03.193933] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49201: ping timer event already removed >[2018-08-18 11:35:03.193951] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-15: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.193963] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-16: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.193999] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c013160, vol0-client-16 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.193993] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-17: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.194015] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c013160, vol0-client-17 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.194015] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c013160, vol0-client-15 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.194134] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49202: ping timer event already removed >[2018-08-18 11:35:03.194218] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.194351] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49202: ping timer event already removed >[2018-08-18 11:35:03.194392] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.194493] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49202: ping timer event already removed >[2018-08-18 11:35:03.194593] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.194639] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.194668] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49203: ping timer event already removed >[2018-08-18 11:35:03.194669] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on vol0-replicate-14 returned error [No such file or directory] >[2018-08-18 11:35:03.194742] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.194808] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49203: ping timer event already removed >[2018-08-18 11:35:03.194902] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.194994] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49203: ping timer event already removed >[2018-08-18 11:35:03.195042] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.195058] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.195066] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on vol0-replicate-15 returned error [No such file or directory] >[2018-08-18 11:35:03.195187] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49204: ping timer event already removed >[2018-08-18 11:35:03.195374] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49204: ping timer event already removed >[2018-08-18 11:35:03.195513] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49204: ping timer event already removed >[2018-08-18 11:35:03.195696] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49205: ping timer event already removed >[2018-08-18 11:35:03.195766] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.195871] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49205: ping timer event already removed >[2018-08-18 11:35:03.195920] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.196067] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49205: ping timer event already removed >[2018-08-18 11:35:03.196139] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.196173] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.196194] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on vol0-replicate-17 returned error [No such file or directory] >[2018-08-18 11:35:03.196270] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49206: ping timer event already removed >[2018-08-18 11:35:03.196335] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.196457] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49206: ping timer event already removed >[2018-08-18 11:35:03.196505] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.196640] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49206: ping timer event already removed >[2018-08-18 11:35:03.196706] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.196723] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.196739] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on vol0-replicate-18 returned error [No such file or directory] >[2018-08-18 11:35:03.196874] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49207: ping timer event already removed >[2018-08-18 11:35:03.196922] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.197066] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49207: ping timer event already removed >[2018-08-18 11:35:03.197111] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.197268] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49207: ping timer event already removed >[2018-08-18 11:35:03.197483] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49208: ping timer event already removed >[2018-08-18 11:35:03.197641] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49208: ping timer event already removed >[2018-08-18 11:35:03.197693] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.197834] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49208: ping timer event already removed >[2018-08-18 11:35:03.197864] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.198056] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49209: ping timer event already removed >[2018-08-18 11:35:03.198103] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.198237] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49209: ping timer event already removed >[2018-08-18 11:35:03.198428] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49209: ping timer event already removed >[2018-08-18 11:35:03.198451] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.198692] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.198696] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49211: ping timer event already removed >[2018-08-18 11:35:03.198761] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9800bcb0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.198811] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9800bcb0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.199001] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49211: ping timer event already removed >[2018-08-18 11:35:03.199040] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.199212] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49211: ping timer event already removed >[2018-08-18 11:35:03.199219] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-63: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.199235] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9800bcb0, vol0-client-63 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.199254] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-64: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.199262] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9800bcb0, vol0-client-64 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.199275] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-65: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.199271] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.199284] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9800bcb0, vol0-client-65 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.199487] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49212: ping timer event already removed >[2018-08-18 11:35:03.199541] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.199698] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c001f40, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.199709] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa40013e0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.199729] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c001f40, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.200077] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49212: ping timer event already removed >[2018-08-18 11:35:03.200219] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-57: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.200225] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-69: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.200245] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c001f40, vol0-client-57 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.200251] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbfa40013e0, vol0-client-69 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.200266] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-59: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.200276] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c001f40, vol0-client-59 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.200268] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-70: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.200270] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-71: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.200271] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-58: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.200294] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbfa40013e0, vol0-client-71 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.200292] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbfa40013e0, vol0-client-70 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.200309] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c001f40, vol0-client-58 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.200370] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9800aea0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.200703] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49212: ping timer event already removed >[2018-08-18 11:35:03.200989] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49213: ping timer event already removed >[2018-08-18 11:35:03.201017] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-61: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.201012] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-60: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.201037] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9800aea0, vol0-client-61 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.201038] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9800aea0, vol0-client-60 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.201051] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-62: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.201085] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf98009700, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.201082] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9800aea0, vol0-client-62 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.201275] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49213: ping timer event already removed >[2018-08-18 11:35:03.201321] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.201322] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa4008720, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.201563] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-72: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.201575] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-73: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.201583] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf98009700, vol0-client-72 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.201589] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf98009700, vol0-client-73 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.201586] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-74: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.201625] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf98009700, vol0-client-74 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.201736] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-66: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.201748] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbfa4008720, vol0-client-66 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.201749] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49213: ping timer event already removed >[2018-08-18 11:35:03.201767] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-68: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.201776] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbfa4008720, vol0-client-68 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.201772] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-67: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.201792] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbfa4008720, vol0-client-67 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.202112] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c00aea0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.202360] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-75: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.202380] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-76: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.202392] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-77: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.202395] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c00aea0, vol0-client-75 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.202409] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c00aea0, vol0-client-77 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.202398] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c00aea0, vol0-client-76 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.202445] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49214: ping timer event already removed >[2018-08-18 11:35:03.202935] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49214: ping timer event already removed >[2018-08-18 11:35:03.203447] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49214: ping timer event already removed >[2018-08-18 11:35:03.203890] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49215: ping timer event already removed >[2018-08-18 11:35:03.204204] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49215: ping timer event already removed >[2018-08-18 11:35:03.204507] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49215: ping timer event already removed >[2018-08-18 11:35:03.204874] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49216: ping timer event already removed >[2018-08-18 11:35:03.205228] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49216: ping timer event already removed >[2018-08-18 11:35:03.205262] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.205480] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49216: ping timer event already removed >[2018-08-18 11:35:03.205809] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa4012390, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.206237] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49217: ping timer event already removed >[2018-08-18 11:35:03.206473] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-84: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.206492] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbfa4012390, vol0-client-84 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.206492] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-86: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.206495] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49217: ping timer event already removed >[2018-08-18 11:35:03.206524] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbfa4012390, vol0-client-86 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.206497] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-85: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.206553] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbfa4012390, vol0-client-85 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.207092] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49217: ping timer event already removed >[2018-08-18 11:35:03.207456] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49218: ping timer event already removed >[2018-08-18 11:35:03.207516] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.207811] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49218: ping timer event already removed >[2018-08-18 11:35:03.208131] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49218: ping timer event already removed >[2018-08-18 11:35:03.208507] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49219: ping timer event already removed >[2018-08-18 11:35:03.208959] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49219: ping timer event already removed >[2018-08-18 11:35:03.209014] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.209402] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49219: ping timer event already removed >[2018-08-18 11:35:03.209865] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49221: ping timer event already removed >[2018-08-18 11:35:03.209906] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.210257] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49221: ping timer event already removed >[2018-08-18 11:35:03.210650] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49221: ping timer event already removed >[2018-08-18 11:35:03.210707] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.211086] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf900015f0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.211111] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf900015f0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.211413] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-99: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.211424] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf900015f0, vol0-client-99 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.211428] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49222: ping timer event already removed >[2018-08-18 11:35:03.211439] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-100: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.211456] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf900015f0, vol0-client-100 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.211458] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-101: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.211474] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf900015f0, vol0-client-101 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.211741] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49222: ping timer event already removed >[2018-08-18 11:35:03.211995] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49222: ping timer event already removed >[2018-08-18 11:35:03.212294] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49223: ping timer event already removed >[2018-08-18 11:35:03.212869] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49223: ping timer event already removed >[2018-08-18 11:35:03.213142] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49223: ping timer event already removed >[2018-08-18 11:35:03.217271] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-vol0-dht: Found anomalies in /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m (gfid = 8faaccaa-0b94-4e58-9a97-0e62c44bd577). Holes=1 overlaps=0 >[2018-08-18 11:35:03.217288] D [MSGID: 0] [dht-common.c:943:dht_lookup_dir_cbk] 0-vol0-dht: fixing assignment on /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m >[2018-08-18 11:35:03.217330] D [MSGID: 0] [dht-selfheal.c:1879:dht_selfheal_layout_new_directory] 0-vol0-dht: chunk size = 0xffffffff / 274638528 = 15.638619 >[2018-08-18 11:35:03.217343] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-28 >[2018-08-18 11:35:03.217349] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-29 >[2018-08-18 11:35:03.217355] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-3 >[2018-08-18 11:35:03.217362] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-30 >[2018-08-18 11:35:03.217370] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-31 >[2018-08-18 11:35:03.217379] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-32 >[2018-08-18 11:35:03.217387] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-33 >[2018-08-18 11:35:03.217395] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-34 >[2018-08-18 11:35:03.217403] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-35 >[2018-08-18 11:35:03.217411] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-4 >[2018-08-18 11:35:03.217420] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-5 >[2018-08-18 11:35:03.217428] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-6 >[2018-08-18 11:35:03.217437] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-7 >[2018-08-18 11:35:03.217445] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-8 >[2018-08-18 11:35:03.217453] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-9 >[2018-08-18 11:35:03.217468] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-0 >[2018-08-18 11:35:03.217477] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-1 >[2018-08-18 11:35:03.217485] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-10 >[2018-08-18 11:35:03.217493] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-11 >[2018-08-18 11:35:03.217499] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-12 >[2018-08-18 11:35:03.217504] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-13 >[2018-08-18 11:35:03.217509] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-14 >[2018-08-18 11:35:03.217515] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-15 >[2018-08-18 11:35:03.217520] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-16 >[2018-08-18 11:35:03.217525] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-17 >[2018-08-18 11:35:03.217530] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-18 >[2018-08-18 11:35:03.217535] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-19 >[2018-08-18 11:35:03.217540] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-2 >[2018-08-18 11:35:03.217545] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-20 >[2018-08-18 11:35:03.217551] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-21 >[2018-08-18 11:35:03.217556] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-22 >[2018-08-18 11:35:03.217561] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-23 >[2018-08-18 11:35:03.217566] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-24 >[2018-08-18 11:35:03.217571] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-25 >[2018-08-18 11:35:03.217576] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-26 >[2018-08-18 11:35:03.217581] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-27 >[2018-08-18 11:35:03.217850] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-2: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.217851] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-1: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.217889] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbf9001c010, vol0-client-2 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.217890] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbf9001c010, vol0-client-1 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.217942] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-0: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.217962] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbf9001c010, vol0-client-0 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.234289] D [MSGID: 0] [dht-selfheal.c:1206:dht_selfheal_dir_mkdir_setacl] 0-vol0-dht: ACL_DEFAULT xattr not present >[2018-08-18 11:35:03.234306] D [MSGID: 0] [dht-selfheal.c:1219:dht_selfheal_dir_mkdir_setacl] 0-vol0-dht: ACL_ACCESS xattr not present >[2018-08-18 11:35:03.234314] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on subvol vol0-replicate-0 >[2018-08-18 11:35:03.234376] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on subvol vol0-replicate-1 >[2018-08-18 11:35:03.234419] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on subvol vol0-replicate-10 >[2018-08-18 11:35:03.234460] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on subvol vol0-replicate-11 >[2018-08-18 11:35:03.234482] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-0: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.234502] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on subvol vol0-replicate-12 >[2018-08-18 11:35:03.234522] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-1: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.234547] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on subvol vol0-replicate-14 >[2018-08-18 11:35:03.234562] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-10: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.234591] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-0: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.234596] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-1: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.234635] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on subvol vol0-replicate-15 >[2018-08-18 11:35:03.234632] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4075730, vol0-client-0 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.234639] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-11: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.234649] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4075730, vol0-client-1 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.234670] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-12: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.234672] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-2: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.234700] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on subvol vol0-replicate-17 >[2018-08-18 11:35:03.234700] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4075730, vol0-client-2 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.234702] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-3: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.234700] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-30: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.234729] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 11:35:03.234733] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd404ff80, vol0-client-30 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.234729] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd405f2c0, vol0-client-3 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.234762] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-31: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.234771] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on subvol vol0-replicate-18 >[2018-08-18 11:35:03.234763] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-5: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.234783] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd405f2c0, vol0-client-5 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.234762] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-4: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.234785] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd404ff80, vol0-client-31 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.234802] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-33: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.234804] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-14: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.234793] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-32: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.234802] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd405f2c0, vol0-client-4 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.234827] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd404ff80, vol0-client-32 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.234847] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4055510, vol0-client-33 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.234850] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-34: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.234847] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on subvol vol0-replicate-2 >[2018-08-18 11:35:03.234852] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 11:35:03.234867] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4055510, vol0-client-34 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.234852] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:03.234868] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-36: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.234893] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4023a50, vol0-client-36 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.234895] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-15: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.234911] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-37: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.234895] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-35: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.234929] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4023a50, vol0-client-37 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.234932] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on subvol vol0-replicate-3 >[2018-08-18 11:35:03.234927] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4055510, vol0-client-35 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.234951] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:03.234959] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-38: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.234970] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-17: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.234984] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on subvol vol0-replicate-4 >[2018-08-18 11:35:03.234971] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4023a50, vol0-client-38 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.234970] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-42: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.234977] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-0 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235012] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4025c90, vol0-client-42 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235013] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 11:35:03.235030] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-43: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235025] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m, gfid = 8faaccaa-0b94-4e58-9a97-0e62c44bd577 [File exists] >[2018-08-18 11:35:03.235041] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4025c90, vol0-client-43 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235039] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-18: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.235057] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on subvol vol0-replicate-6 >[2018-08-18 11:35:03.235055] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-44: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235084] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4025c90, vol0-client-44 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235089] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-45: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235090] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-2: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.235100] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4023240, vol0-client-45 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235109] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 11:35:03.235115] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-1 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235116] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-46: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235130] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on subvol vol0-replicate-8 >[2018-08-18 11:35:03.235116] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-47: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235133] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m, gfid = 8faaccaa-0b94-4e58-9a97-0e62c44bd577 [File exists] >[2018-08-18 11:35:03.235139] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4023240, vol0-client-46 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235162] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4023240, vol0-client-47 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235177] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 11:35:03.235177] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-10 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235187] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-3: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.235199] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-51: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235206] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m on subvol vol0-replicate-9 >[2018-08-18 11:35:03.235208] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m, gfid = 8faaccaa-0b94-4e58-9a97-0e62c44bd577 [File exists] >[2018-08-18 11:35:03.235206] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-52: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235214] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd40087d0, vol0-client-51 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235238] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd40087d0, vol0-client-52 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235244] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-11 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235260] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-53: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235268] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-4: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.235268] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m, gfid = 8faaccaa-0b94-4e58-9a97-0e62c44bd577 [File exists] >[2018-08-18 11:35:03.235271] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-54: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235311] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4005560, vol0-client-54 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235297] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-55: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235318] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-12 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235333] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-6: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235336] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-56: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235315] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd40087d0, vol0-client-53 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235333] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4005560, vol0-client-55 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235358] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4002d40, vol0-client-6 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235363] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 11:35:03.235367] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-6: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.235343] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m, gfid = 8faaccaa-0b94-4e58-9a97-0e62c44bd577 [File exists] >[2018-08-18 11:35:03.235396] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4005560, vol0-client-56 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235399] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-8: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235401] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-7: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235414] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-14 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235414] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4002d40, vol0-client-8 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235419] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4002d40, vol0-client-7 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235420] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 11:35:03.235426] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m, gfid = 8faaccaa-0b94-4e58-9a97-0e62c44bd577 [File exists] >[2018-08-18 11:35:03.235436] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 11:35:03.235441] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-9: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235443] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-8: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.235455] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-15 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235455] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd40213d0, vol0-client-9 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235483] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-10: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235488] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m, gfid = 8faaccaa-0b94-4e58-9a97-0e62c44bd577 [File exists] >[2018-08-18 11:35:03.235484] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-11: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235496] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd40213d0, vol0-client-10 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235511] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd40213d0, vol0-client-11 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235511] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-12: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235521] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-13: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235514] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-9: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.235529] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-14: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235545] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 11:35:03.235548] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4066240, vol0-client-13 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235551] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4066240, vol0-client-12 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235543] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4066240, vol0-client-14 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235544] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-18: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235578] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-20: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235580] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4075730, vol0-client-18 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235581] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:03.235579] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-19: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235616] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-18 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235632] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4075730, vol0-client-19 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235633] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-24: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235620] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4075730, vol0-client-20 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235642] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m, gfid = 8faaccaa-0b94-4e58-9a97-0e62c44bd577 [File exists] >[2018-08-18 11:35:03.235643] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-17 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235649] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4042e10, vol0-client-24 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235650] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-2 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235664] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m, gfid = 8faaccaa-0b94-4e58-9a97-0e62c44bd577 [File exists] >[2018-08-18 11:35:03.235670] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m, gfid = 8faaccaa-0b94-4e58-9a97-0e62c44bd577 [File exists] >[2018-08-18 11:35:03.235664] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 11:35:03.235681] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-26: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235681] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-25: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235695] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4042e10, vol0-client-26 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235699] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd4042e10, vol0-client-25 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235703] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-29: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235701] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-28: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235681] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-27: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m [File exists] >[2018-08-18 11:35:03.235740] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd405f2c0, vol0-client-28 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235736] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 11:35:03.235752] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-3 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235748] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd405f2c0, vol0-client-29 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235735] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd405f2c0, vol0-client-27 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235767] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m, gfid = 8faaccaa-0b94-4e58-9a97-0e62c44bd577 [File exists] >[2018-08-18 11:35:03.235762] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-4 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235794] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 11:35:03.235795] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m, gfid = 8faaccaa-0b94-4e58-9a97-0e62c44bd577 [File exists] >[2018-08-18 11:35:03.235822] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-6 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235833] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m, gfid = 8faaccaa-0b94-4e58-9a97-0e62c44bd577 [File exists] >[2018-08-18 11:35:03.235881] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-8 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235899] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m, gfid = 8faaccaa-0b94-4e58-9a97-0e62c44bd577 [File exists] >[2018-08-18 11:35:03.235942] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa0003040, vol0-replicate-9 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.235957] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m, gfid = 8faaccaa-0b94-4e58-9a97-0e62c44bd577 [File exists] >[2018-08-18 11:35:03.237636] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-0: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.237656] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-1: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.237668] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.237700] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.237733] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.237803] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-12: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.237826] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 11:35:03.237846] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 11:35:03.237868] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.237896] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 11:35:03.237890] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:03.237927] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:03.237966] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 11:35:03.237972] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.238040] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-15: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.238079] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 11:35:03.238080] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-19: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.238170] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-17: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.238190] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 11:35:03.238202] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-20: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.238264] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.238269] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 11:35:03.238292] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 11:35:03.238319] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-18: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.238335] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-22: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.238352] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 11:35:03.238411] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-23: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.238431] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-2: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.238466] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-20: lk op is for a transaction >[2018-08-18 11:35:03.238496] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 11:35:03.238525] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.238538] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-22: lk op is for a transaction >[2018-08-18 11:35:03.238563] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 11:35:03.238575] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-25: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.238622] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 11:35:03.238647] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-23: lk op is for a transaction >[2018-08-18 11:35:03.238727] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:03.238729] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-26: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.238780] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-27: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.238820] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 11:35:03.238898] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-28: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.238939] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-26: lk op is for a transaction >[2018-08-18 11:35:03.238982] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-29: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.239008] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-27: lk op is for a transaction >[2018-08-18 11:35:03.239102] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 11:35:03.239119] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-30: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.239158] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.239200] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 11:35:03.239203] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-32: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.239235] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-33: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.239256] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-3: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.239307] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-34: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.239322] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-30: lk op is for a transaction >[2018-08-18 11:35:03.239368] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 11:35:03.239369] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-35: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.239390] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 11:35:03.239422] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 11:35:03.239472] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 11:35:03.239482] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 11:35:03.239531] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-5: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.239566] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 11:35:03.239647] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.239664] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.239765] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 11:35:03.239781] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-6: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.239817] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 11:35:03.239865] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:03.239930] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-8: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.239946] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 11:35:03.240001] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-9: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.240106] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 11:35:03.240243] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 11:35:03.240321] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-0: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.240334] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd805b530, vol0-client-0 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.240337] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-1: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.240351] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd805b530, vol0-client-1 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.240351] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-2: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.240371] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd805b530, vol0-client-2 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.294982] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88 on vol0-replicate-6 >[2018-08-18 11:35:03.295257] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295291] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295308] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295323] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295338] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88 with op_ret -1 [No such file or directory] >[2018-08-18 11:35:03.295346] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88 missing on subvol vol0-replicate-6 >[2018-08-18 11:35:03.295353] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:03.295567] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295574] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295580] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295607] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295620] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-0 >[2018-08-18 11:35:03.295621] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295627] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295658] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295665] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295682] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295694] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295698] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295703] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-2 >[2018-08-18 11:35:03.295715] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295722] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295724] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295770] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295773] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295779] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295774] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295780] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295803] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-3 >[2018-08-18 11:35:03.295804] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-1 >[2018-08-18 11:35:03.295811] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295831] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295831] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-4 >[2018-08-18 11:35:03.295846] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295860] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295876] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295887] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-5 >[2018-08-18 11:35:03.295928] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295950] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295958] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.295981] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296015] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296034] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296051] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296062] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296073] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-7 >[2018-08-18 11:35:03.296077] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296087] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296098] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296087] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296112] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296128] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-8 >[2018-08-18 11:35:03.296128] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296128] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296153] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296160] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-9 >[2018-08-18 11:35:03.296170] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296200] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296203] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296209] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296218] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296222] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-10 >[2018-08-18 11:35:03.296230] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296239] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-11 >[2018-08-18 11:35:03.296254] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296258] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296268] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296285] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296292] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296297] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-12 >[2018-08-18 11:35:03.296318] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296326] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296341] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296353] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-13 >[2018-08-18 11:35:03.296387] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296400] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296425] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296442] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296453] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-14 >[2018-08-18 11:35:03.296464] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296489] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296489] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296518] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296528] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-15 >[2018-08-18 11:35:03.296532] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296543] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296555] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296571] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296579] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-16 >[2018-08-18 11:35:03.296589] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296593] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296589] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296636] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296641] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296638] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296637] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296675] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296676] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-17 >[2018-08-18 11:35:03.296687] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-18 >[2018-08-18 11:35:03.296693] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296708] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296717] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296743] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296745] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296757] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-19 >[2018-08-18 11:35:03.296757] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296757] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296788] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296798] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-20 >[2018-08-18 11:35:03.296793] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296817] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296827] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296848] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296852] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296865] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-21 >[2018-08-18 11:35:03.296869] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296882] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296895] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296904] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-22 >[2018-08-18 11:35:03.296928] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296935] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296955] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296971] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.296979] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-23 >[2018-08-18 11:35:03.297009] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297023] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297053] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297067] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297075] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-24 >[2018-08-18 11:35:03.297090] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297096] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297115] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297131] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297141] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-25 >[2018-08-18 11:35:03.297142] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297141] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297197] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297200] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297197] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297226] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297231] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297240] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297251] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-26 >[2018-08-18 11:35:03.297256] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297252] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297256] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297271] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-27 >[2018-08-18 11:35:03.297290] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297301] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297302] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-28 >[2018-08-18 11:35:03.297302] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297318] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297349] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297349] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297351] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297350] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297375] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-29 >[2018-08-18 11:35:03.297399] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297401] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297405] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297417] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-30 >[2018-08-18 11:35:03.297438] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297444] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297442] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297459] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297460] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297471] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-31 >[2018-08-18 11:35:03.297477] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297485] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-32 >[2018-08-18 11:35:03.297533] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297542] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297562] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297574] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297575] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297595] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-33 >[2018-08-18 11:35:03.297596] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297619] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-6 >[2018-08-18 11:35:03.297621] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297658] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297663] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297687] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297692] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297691] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297706] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297709] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-34 >[2018-08-18 11:35:03.297727] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297736] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-35 >[2018-08-18 11:35:03.297744] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-6 cached_subvol null >[2018-08-18 11:35:03.297751] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88 >[2018-08-18 11:35:03.297758] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297765] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297773] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297780] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.297788] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80031b0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.298078] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-0': avail_percent is: 96.00 and avail_space is: 7682560311296 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.298116] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-1': avail_percent is: 95.00 and avail_space is: 7649225949184 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.298147] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-2': avail_percent is: 96.00 and avail_space is: 7679637286912 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.298195] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-3': avail_percent is: 95.00 and avail_space is: 7676722438144 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.298243] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-4': avail_percent is: 96.00 and avail_space is: 7679894794240 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.298281] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-5': avail_percent is: 95.00 and avail_space is: 7672041328640 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.298359] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-6': avail_percent is: 94.00 and avail_space is: 7567703506944 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.298420] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-7': avail_percent is: 95.00 and avail_space is: 7599553392640 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.298450] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-8': avail_percent is: 95.00 and avail_space is: 7669932085248 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.298492] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-9': avail_percent is: 95.00 and avail_space is: 7675189104640 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.298525] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-10': avail_percent is: 94.00 and avail_space is: 7587263074304 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.298586] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-11': avail_percent is: 95.00 and avail_space is: 7666176073728 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.298624] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-12': avail_percent is: 95.00 and avail_space is: 7662328016896 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.298709] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-13': avail_percent is: 95.00 and avail_space is: 7671819669504 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.298732] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-14': avail_percent is: 95.00 and avail_space is: 7665779671040 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.298770] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-15': avail_percent is: 95.00 and avail_space is: 7654391312384 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.298857] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-16': avail_percent is: 95.00 and avail_space is: 7678815076352 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.298927] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-17': avail_percent is: 95.00 and avail_space is: 7669480648704 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.298946] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-18': avail_percent is: 95.00 and avail_space is: 7615159742464 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.298988] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-19': avail_percent is: 95.00 and avail_space is: 7660373098496 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.299025] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-20': avail_percent is: 95.00 and avail_space is: 7610263240704 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.299077] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-21': avail_percent is: 95.00 and avail_space is: 7663388852224 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.299111] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-22': avail_percent is: 95.00 and avail_space is: 7668082892800 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.299156] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-23': avail_percent is: 95.00 and avail_space is: 7669669240832 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.299206] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-24': avail_percent is: 95.00 and avail_space is: 7660816109568 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.299253] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-25': avail_percent is: 95.00 and avail_space is: 7644139499520 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.299323] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-26': avail_percent is: 95.00 and avail_space is: 7655957176320 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.299354] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-27': avail_percent is: 95.00 and avail_space is: 7652166385664 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.299396] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-28': avail_percent is: 95.00 and avail_space is: 7652181016576 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.299436] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-29': avail_percent is: 95.00 and avail_space is: 7639365349376 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.299475] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-30': avail_percent is: 95.00 and avail_space is: 7675637047296 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.299525] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-31': avail_percent is: 95.00 and avail_space is: 7678164856832 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.299565] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-32': avail_percent is: 95.00 and avail_space is: 7659078995968 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.299630] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-33': avail_percent is: 95.00 and avail_space is: 7676539994112 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.299654] D [MSGID: 0] [dht-common.c:6923:dht_create_wind_to_avail_subvol] 0-vol0-dht: creating /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88 on vol0-replicate-6 >[2018-08-18 11:35:03.299689] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-34': avail_percent is: 95.00 and avail_space is: 7665343188992 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.299726] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-35': avail_percent is: 95.00 and avail_space is: 7661847982080 and avail_inodes is: 99.00 >[2018-08-18 11:35:03.299890] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-6: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.300150] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-6 >[2018-08-18 11:35:03.300179] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 11:35:03.462360] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7fbfa0015d20) as it exceeds the maximum available buffer size >[2018-08-18 11:35:03.463307] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:03.463332] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=d72239d0-643a-4fd7-9483-ce8957d4000d, gen=0): picking the request for winding >[2018-08-18 11:35:03.463343] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=d72239d0-643a-4fd7-9483-ce8957d4000d, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:03.463557] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-6: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.482108] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=d72239d0-643a-4fd7-9483-ce8957d4000d, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:03.482330] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=d72239d0-643a-4fd7-9483-ce8957d4000d, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:03.482523] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:03.622271] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7fbfa802ae80) as it exceeds the maximum available buffer size >[2018-08-18 11:35:03.623559] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:03.623592] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=d72239d0-643a-4fd7-9483-ce8957d4000d, gen=0): picking the request for winding >[2018-08-18 11:35:03.623619] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=d72239d0-643a-4fd7-9483-ce8957d4000d, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:03.623692] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-6: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.623710] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 11:35:03.623738] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7fbfa0015d20) allocated with standard calloc() >[2018-08-18 11:35:03.635395] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:03.635438] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=1, gfid=d72239d0-643a-4fd7-9483-ce8957d4000d. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=d72239d0-643a-4fd7-9483-ce8957d4000d, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:03.640289] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=d72239d0-643a-4fd7-9483-ce8957d4000d, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:03.640447] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=d72239d0-643a-4fd7-9483-ce8957d4000d, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:03.640579] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:03.640609] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=d72239d0-643a-4fd7-9483-ce8957d4000d, gen=1): picking the request for winding >[2018-08-18 11:35:03.640887] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=d72239d0-643a-4fd7-9483-ce8957d4000d, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:03.640968] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 11:35:03.641152] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7fbfa802ae80) allocated with standard calloc() >[2018-08-18 11:35:03.641905] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001 on vol0-replicate-5 >[2018-08-18 11:35:03.642200] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.642228] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.642238] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.642270] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.642285] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001 with op_ret -1 [No such file or directory] >[2018-08-18 11:35:03.642297] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001 missing on subvol vol0-replicate-5 >[2018-08-18 11:35:03.642305] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:03.642596] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.642634] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.642647] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.642676] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.642681] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.642702] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-0 >[2018-08-18 11:35:03.642720] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.642754] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.642728] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.642767] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.642803] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.642807] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.642838] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.642841] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-1 >[2018-08-18 11:35:03.642863] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.642863] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-2 >[2018-08-18 11:35:03.642892] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.642973] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643017] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643013] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643014] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643031] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643041] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643044] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643045] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-3 >[2018-08-18 11:35:03.643071] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643085] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-4 >[2018-08-18 11:35:03.643124] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643182] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643192] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643218] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643236] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-6 >[2018-08-18 11:35:03.643266] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643282] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643292] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643305] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643314] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-7 >[2018-08-18 11:35:03.643366] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643399] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643410] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643425] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643429] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643450] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643469] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-8 >[2018-08-18 11:35:03.643489] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643501] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643518] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-9 >[2018-08-18 11:35:03.643518] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643524] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643526] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643560] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643575] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-10 >[2018-08-18 11:35:03.643590] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643610] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643619] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643648] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643650] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643666] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-11 >[2018-08-18 11:35:03.643686] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643680] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643708] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643724] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-12 >[2018-08-18 11:35:03.643761] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643793] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643848] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643871] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643886] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-13 >[2018-08-18 11:35:03.643902] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643904] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643948] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643974] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643985] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.643996] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-14 >[2018-08-18 11:35:03.644024] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644032] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644049] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644070] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644084] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644070] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644099] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-15 >[2018-08-18 11:35:03.644128] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644141] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644157] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-16 >[2018-08-18 11:35:03.644164] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644168] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644194] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644212] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-17 >[2018-08-18 11:35:03.644212] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644228] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644243] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644255] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644263] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-18 >[2018-08-18 11:35:03.644278] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644280] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644325] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644350] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644364] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-19 >[2018-08-18 11:35:03.644392] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644397] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644409] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644447] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644465] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-20 >[2018-08-18 11:35:03.644470] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644490] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644517] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644541] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644556] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-21 >[2018-08-18 11:35:03.644556] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644562] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644594] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644621] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644630] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-22 >[2018-08-18 11:35:03.644666] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644679] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644711] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644726] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644733] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-23 >[2018-08-18 11:35:03.644794] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644826] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644834] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644860] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644876] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-24 >[2018-08-18 11:35:03.644881] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644899] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644922] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644943] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644957] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.644958] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-25 >[2018-08-18 11:35:03.644977] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645041] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645048] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645071] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645080] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645089] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-26 >[2018-08-18 11:35:03.645104] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645122] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645134] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-27 >[2018-08-18 11:35:03.645178] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645179] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645232] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645251] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645256] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645264] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-28 >[2018-08-18 11:35:03.645278] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645334] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645347] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645352] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645355] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645369] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-29 >[2018-08-18 11:35:03.645378] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645405] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645421] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-30 >[2018-08-18 11:35:03.645425] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645458] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645460] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645494] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645503] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645507] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645508] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-31 >[2018-08-18 11:35:03.645538] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645550] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645558] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-32 >[2018-08-18 11:35:03.645606] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645640] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645641] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645656] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645664] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-33 >[2018-08-18 11:35:03.645690] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645711] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645722] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645733] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-5 >[2018-08-18 11:35:03.645744] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645749] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645788] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645803] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-34 >[2018-08-18 11:35:03.645847] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645850] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645870] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645889] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645898] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-35 >[2018-08-18 11:35:03.645910] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-5 cached_subvol null >[2018-08-18 11:35:03.645918] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001 >[2018-08-18 11:35:03.645926] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x2029970, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645935] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645945] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645955] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.645964] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.646613] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m returned with op_ret 0 >[2018-08-18 11:35:03.650035] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:03.650062] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m returned with op_ret 0" repeated 35 times between [2018-08-18 11:35:03.646613] and [2018-08-18 11:35:03.649683] >[2018-08-18 11:35:03.650061] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=d72239d0-643a-4fd7-9483-ce8957d4000d, gen=0): picking the request for winding >[2018-08-18 11:35:03.650108] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88 at vol0-replicate-6 >[2018-08-18 11:35:03.650335] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=d72239d0-643a-4fd7-9483-ce8957d4000d, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:03.650418] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88 returned with op_ret 0 >[2018-08-18 11:35:03.650563] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001 on vol0-replicate-5 >[2018-08-18 11:35:03.650791] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.650822] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.650825] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.650843] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.650851] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001 with op_ret -1 [No such file or directory] >[2018-08-18 11:35:03.650859] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001 missing on subvol vol0-replicate-5 >[2018-08-18 11:35:03.650866] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:03.651083] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651113] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651110] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651139] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651163] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651177] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-0 >[2018-08-18 11:35:03.651175] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651178] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651212] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651221] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-1 >[2018-08-18 11:35:03.651232] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651236] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651249] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651235] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651268] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651272] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651271] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651309] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-2 >[2018-08-18 11:35:03.651307] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651309] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651327] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651327] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-3 >[2018-08-18 11:35:03.651328] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651361] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651373] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651388] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651391] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-4 >[2018-08-18 11:35:03.651438] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651459] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651468] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651486] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651495] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-6 >[2018-08-18 11:35:03.651518] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651540] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651563] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651572] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651580] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-7 >[2018-08-18 11:35:03.651620] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651625] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651663] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651671] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651691] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651709] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651714] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651727] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-8 >[2018-08-18 11:35:03.651736] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651740] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651757] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651762] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651765] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-10 >[2018-08-18 11:35:03.651781] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651791] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-9 >[2018-08-18 11:35:03.651793] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651811] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651832] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651845] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651851] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651873] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651875] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651896] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-11 >[2018-08-18 11:35:03.651897] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651918] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-12 >[2018-08-18 11:35:03.651942] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651952] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651945] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651983] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.651987] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652005] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-13 >[2018-08-18 11:35:03.652009] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652027] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652050] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652070] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-14 >[2018-08-18 11:35:03.652082] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652071] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652092] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652150] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652156] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652156] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652144] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652187] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652180] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-15 >[2018-08-18 11:35:03.652181] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652211] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652229] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652236] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652236] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652238] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652254] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-16 >[2018-08-18 11:35:03.652264] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-17 >[2018-08-18 11:35:03.652275] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652289] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652304] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652301] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652300] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652329] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652335] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-18 >[2018-08-18 11:35:03.652336] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652346] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652358] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652360] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-19 >[2018-08-18 11:35:03.652372] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652381] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-20 >[2018-08-18 11:35:03.652404] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652407] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652427] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652457] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652462] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652477] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-21 >[2018-08-18 11:35:03.652485] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652502] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652521] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652530] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-22 >[2018-08-18 11:35:03.652560] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652595] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652614] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652631] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652634] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652654] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-23 >[2018-08-18 11:35:03.652653] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652677] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652679] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652697] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652704] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652715] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652722] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-24 >[2018-08-18 11:35:03.652736] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652750] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-25 >[2018-08-18 11:35:03.652754] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652776] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652790] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652795] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652806] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652809] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652820] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-26 >[2018-08-18 11:35:03.652840] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652850] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652857] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-27 >[2018-08-18 11:35:03.652860] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652868] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652870] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652889] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652901] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-28 >[2018-08-18 11:35:03.652914] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652936] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.652996] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653016] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653019] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653032] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653040] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-30 >[2018-08-18 11:35:03.653052] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653061] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653074] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653069] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653083] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-29 >[2018-08-18 11:35:03.653069] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653100] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653113] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653113] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-31 >[2018-08-18 11:35:03.653127] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653137] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653153] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653160] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653170] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653165] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653172] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653195] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-32 >[2018-08-18 11:35:03.653214] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653214] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653216] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653239] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-33 >[2018-08-18 11:35:03.653249] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653250] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653255] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-5 >[2018-08-18 11:35:03.653269] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653284] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-34 >[2018-08-18 11:35:03.653295] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653298] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653318] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653335] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653346] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001) from subvol vol0-replicate-35 >[2018-08-18 11:35:03.653359] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-5 cached_subvol null >[2018-08-18 11:35:03.653379] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001 >[2018-08-18 11:35:03.653389] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653401] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653414] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653426] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653438] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.653557] I [MSGID: 109066] [dht-rename.c:1610:dht_rename] 0-vol0-dht: renaming /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001_bf5c1e04a2da11e8a3196c92bf5c1b88 (hash=vol0-replicate-6/cache=vol0-replicate-6) => /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001 (hash=vol0-replicate-5/cache=<nul>) >[2018-08-18 11:35:03.654173] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-5: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.654781] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-102: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 11:35:03.654812] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-103: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 11:35:03.654822] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-104: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 11:35:03.681948] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 11:35:03.682054] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-6: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.682347] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 11:35:03.682448] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-5: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.682862] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 11:35:03.682948] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-6: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.683233] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 11:35:03.683307] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-5: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.683486] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 11:35:03.684112] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:03.684156] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=d72239d0-643a-4fd7-9483-ce8957d4000d, gen=0): picking the request for winding >[2018-08-18 11:35:03.684197] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001 at vol0-replicate-6 >[2018-08-18 11:35:03.684517] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001 returned with op_ret 0 >[2018-08-18 11:35:03.684544] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=d72239d0-643a-4fd7-9483-ce8957d4000d, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:03.684868] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-6: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.685080] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 11:35:03.685415] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:03.685448] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=d72239d0-643a-4fd7-9483-ce8957d4000d, gen=0): picking the request for winding >[2018-08-18 11:35:03.685485] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001 at vol0-replicate-6 >[2018-08-18 11:35:03.685846] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=d72239d0-643a-4fd7-9483-ce8957d4000d, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:03.685863] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001 returned with op_ret 0 >[2018-08-18 11:35:03.686154] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-6: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.686351] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 11:35:03.686694] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:03.686735] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=d72239d0-643a-4fd7-9483-ce8957d4000d, gen=0): picking the request for winding >[2018-08-18 11:35:03.686773] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001 at vol0-replicate-6 >[2018-08-18 11:35:03.687104] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001 returned with op_ret 0 >[2018-08-18 11:35:03.687101] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=d72239d0-643a-4fd7-9483-ce8957d4000d, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:03.687407] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-6: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.687629] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 11:35:03.687951] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:03.687982] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=d72239d0-643a-4fd7-9483-ce8957d4000d, gen=0): picking the request for winding >[2018-08-18 11:35:03.688019] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001 at vol0-replicate-6 >[2018-08-18 11:35:03.688336] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=d72239d0-643a-4fd7-9483-ce8957d4000d, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:03.688354] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/0b0849197076340f80ef203f913cef87_171654945114710961.m/0001 returned with op_ret 0 >[2018-08-18 11:35:03.688624] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-6: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.688814] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 11:35:03.704002] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-7 >[2018-08-18 11:35:03.704268] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704271] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704290] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704312] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704322] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m with op_ret -1 [No such file or directory] >[2018-08-18 11:35:03.704330] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m missing on subvol vol0-replicate-7 >[2018-08-18 11:35:03.704336] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:03.704547] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704609] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704612] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704630] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704638] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704656] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704663] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704666] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704687] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704678] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-0 >[2018-08-18 11:35:03.704680] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704721] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704724] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704734] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704741] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-1 >[2018-08-18 11:35:03.704744] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704746] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-2 >[2018-08-18 11:35:03.704780] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704795] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704796] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704803] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-3 >[2018-08-18 11:35:03.704814] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704822] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704842] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704845] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704856] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-4 >[2018-08-18 11:35:03.704860] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704867] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704888] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704896] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-5 >[2018-08-18 11:35:03.704904] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704918] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704948] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704956] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704969] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704976] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.704994] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-6 >[2018-08-18 11:35:03.705034] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705059] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705068] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705086] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705095] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705101] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-8 >[2018-08-18 11:35:03.705127] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705141] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705155] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705169] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-9 >[2018-08-18 11:35:03.705201] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705203] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705237] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705251] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705258] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-10 >[2018-08-18 11:35:03.705286] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705290] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705291] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705311] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705318] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-11 >[2018-08-18 11:35:03.705329] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705347] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705356] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705367] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705375] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-12 >[2018-08-18 11:35:03.705400] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705403] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705409] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705443] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705451] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705449] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705464] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-13 >[2018-08-18 11:35:03.705490] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705517] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705512] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705545] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705545] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705614] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705595] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705590] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-14 >[2018-08-18 11:35:03.705590] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705607] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705669] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-15 >[2018-08-18 11:35:03.705682] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705677] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705680] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705712] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-16 >[2018-08-18 11:35:03.705713] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705730] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705734] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705740] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-17 >[2018-08-18 11:35:03.705767] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705769] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705780] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705790] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705804] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-18 >[2018-08-18 11:35:03.705812] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705844] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705860] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705864] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705878] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-19 >[2018-08-18 11:35:03.705886] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705921] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705950] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705958] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705950] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.705981] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-20 >[2018-08-18 11:35:03.706028] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706037] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706042] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706043] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706034] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706078] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706083] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706090] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706085] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706118] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-22 >[2018-08-18 11:35:03.706123] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706121] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-21 >[2018-08-18 11:35:03.706157] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706159] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-23 >[2018-08-18 11:35:03.706173] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706205] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706206] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706217] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706223] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706219] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706251] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706249] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-24 >[2018-08-18 11:35:03.706249] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706277] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-25 >[2018-08-18 11:35:03.706283] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706286] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706312] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706314] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706317] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706339] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-26 >[2018-08-18 11:35:03.706359] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706365] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706382] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706388] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706406] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706409] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706419] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706415] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-27 >[2018-08-18 11:35:03.706427] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-28 >[2018-08-18 11:35:03.706443] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706465] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706484] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706492] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-29 >[2018-08-18 11:35:03.706508] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706524] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706555] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706569] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706577] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-30 >[2018-08-18 11:35:03.706608] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706633] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706647] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706666] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706672] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706687] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706697] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-32 >[2018-08-18 11:35:03.706700] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706700] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706725] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706730] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706733] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706742] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-31 >[2018-08-18 11:35:03.706738] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-7 >[2018-08-18 11:35:03.706748] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706759] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706761] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706780] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706789] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-33 >[2018-08-18 11:35:03.706801] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706804] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706824] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706833] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706836] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706843] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-34 >[2018-08-18 11:35:03.706844] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706859] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706866] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-35 >[2018-08-18 11:35:03.706875] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-7 cached_subvol null >[2018-08-18 11:35:03.706881] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m >[2018-08-18 11:35:03.706888] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706894] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706902] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706909] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.706917] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.707425] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0 returned with op_ret 0 >[2018-08-18 11:35:03.710871] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0 returned with op_ret 0" repeated 35 times between [2018-08-18 11:35:03.707425] and [2018-08-18 11:35:03.710569] >[2018-08-18 11:35:03.710870] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-7 >[2018-08-18 11:35:03.711135] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711156] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711163] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711179] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711188] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m with op_ret -1 [No such file or directory] >[2018-08-18 11:35:03.711196] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m missing on subvol vol0-replicate-7 >[2018-08-18 11:35:03.711203] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:03.711417] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711434] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711451] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711467] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711472] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711476] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-0 >[2018-08-18 11:35:03.711478] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711491] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711502] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711510] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-1 >[2018-08-18 11:35:03.711530] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711549] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711549] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711569] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711585] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711596] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-2 >[2018-08-18 11:35:03.711604] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711617] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711618] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711620] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711648] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711649] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711668] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-3 >[2018-08-18 11:35:03.711668] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711685] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-4 >[2018-08-18 11:35:03.711682] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711692] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711687] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711724] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711731] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711726] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711733] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711747] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-5 >[2018-08-18 11:35:03.711764] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711767] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711781] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-6 >[2018-08-18 11:35:03.711791] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711843] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711857] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711879] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711888] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711896] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-8 >[2018-08-18 11:35:03.711904] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711925] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711928] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711941] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.711954] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-9 >[2018-08-18 11:35:03.711997] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712012] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712019] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712032] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712039] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-10 >[2018-08-18 11:35:03.712052] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712069] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712076] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712104] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712139] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712145] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712163] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712167] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712177] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712177] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712188] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712195] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712206] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-12 >[2018-08-18 11:35:03.712194] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-11 >[2018-08-18 11:35:03.712225] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712229] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712227] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712241] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-13 >[2018-08-18 11:35:03.712252] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712258] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712267] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712272] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712283] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-14 >[2018-08-18 11:35:03.712290] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712299] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-15 >[2018-08-18 11:35:03.712327] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712380] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712379] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712410] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712422] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712423] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-16 >[2018-08-18 11:35:03.712441] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712451] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712462] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712469] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-17 >[2018-08-18 11:35:03.712486] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712494] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712496] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712516] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712523] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-18 >[2018-08-18 11:35:03.712554] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712555] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712571] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712588] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712603] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-19 >[2018-08-18 11:35:03.712619] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712642] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712642] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712679] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712691] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712693] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-20 >[2018-08-18 11:35:03.712741] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712737] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712766] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712762] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712762] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712791] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712805] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712806] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-21 >[2018-08-18 11:35:03.712816] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-22 >[2018-08-18 11:35:03.712847] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712860] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712881] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712893] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712900] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-23 >[2018-08-18 11:35:03.712929] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712934] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712956] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712971] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712978] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.712978] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-24 >[2018-08-18 11:35:03.712997] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713020] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713024] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713036] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713036] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713044] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-25 >[2018-08-18 11:35:03.713048] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713080] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713077] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713099] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713102] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713099] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-26 >[2018-08-18 11:35:03.713116] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713119] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713136] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-27 >[2018-08-18 11:35:03.713140] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713152] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713175] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713179] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713193] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-28 >[2018-08-18 11:35:03.713197] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713247] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713265] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713261] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713285] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-29 >[2018-08-18 11:35:03.713286] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713297] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713322] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713327] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713340] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-30 >[2018-08-18 11:35:03.713363] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713365] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713365] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713387] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713388] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713400] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713416] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-31 >[2018-08-18 11:35:03.713417] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713433] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713441] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713443] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713440] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713434] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-32 >[2018-08-18 11:35:03.713467] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713475] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713481] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713486] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713497] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-7 >[2018-08-18 11:35:03.713492] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713486] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-33 >[2018-08-18 11:35:03.713527] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713531] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713536] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-34 >[2018-08-18 11:35:03.713555] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713559] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713581] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713588] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m) from subvol vol0-replicate-35 >[2018-08-18 11:35:03.713596] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-7 cached_subvol null >[2018-08-18 11:35:03.713611] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m >[2018-08-18 11:35:03.713618] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713624] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713631] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713638] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.713645] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.714155] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-7: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.734887] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-7 >[2018-08-18 11:35:03.735160] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.735185] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.737143] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9800aea0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.737166] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9800aea0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.737579] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-21: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.737593] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9800aea0, vol0-client-21 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.737614] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-23: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.737616] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-22: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.737641] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9800aea0, vol0-client-23 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.737645] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9800aea0, vol0-client-22 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.744690] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-0: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.744728] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-1: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.744798] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-2: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.744811] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-3: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.744868] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-4: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.744911] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-5: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.744949] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-6: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.745020] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-8: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.745048] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-9: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.745102] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-10: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.745120] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-11: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.745165] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-12: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.745191] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-13: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.745262] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-14: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.745311] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-15: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.745366] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-16: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.745419] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-17: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.745482] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-18: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.745591] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-20: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.745593] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-19: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.745654] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-21: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.745657] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 11:35:03.745693] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-22: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.745770] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-23: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.745838] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-24: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.745892] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-25: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.745948] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-26: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.746010] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-27: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.746074] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-28: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.746148] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-29: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.746223] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-30: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.746265] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-31: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.746326] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-32: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.746362] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-33: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.746397] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 11:35:03.746436] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-34: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.746504] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-35: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.746762] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 11:35:03.748681] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m with op_ret 0 >[2018-08-18 11:35:03.748971] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.748967] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749004] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749030] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749040] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749050] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-0 returned error [No such file or directory] >[2018-08-18 11:35:03.749058] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749074] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749088] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749099] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-1 returned error [No such file or directory] >[2018-08-18 11:35:03.749158] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749199] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749203] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749226] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749238] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-2 returned error [No such file or directory] >[2018-08-18 11:35:03.749366] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749436] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749448] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749474] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749492] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749502] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-4 returned error [No such file or directory] >[2018-08-18 11:35:03.749523] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749540] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749563] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749582] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749591] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-5 returned error [No such file or directory] >[2018-08-18 11:35:03.749655] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749680] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749703] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749719] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749727] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-6 returned error [No such file or directory] >[2018-08-18 11:35:03.749878] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749921] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749946] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749973] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.749998] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-8 returned error [No such file or directory] >[2018-08-18 11:35:03.750027] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750027] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750130] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750169] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750167] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750204] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750222] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-10 returned error [No such file or directory] >[2018-08-18 11:35:03.750282] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750288] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750303] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750329] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750346] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-11 returned error [No such file or directory] >[2018-08-18 11:35:03.750409] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750446] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750478] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750501] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750500] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750537] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-12 returned error [No such file or directory] >[2018-08-18 11:35:03.750560] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750593] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750617] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750627] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-13 returned error [No such file or directory] >[2018-08-18 11:35:03.750647] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750678] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750683] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750706] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750720] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-14 returned error [No such file or directory] >[2018-08-18 11:35:03.750860] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750888] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750894] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750918] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.750931] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-15 returned error [No such file or directory] >[2018-08-18 11:35:03.751049] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751075] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751104] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751126] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751135] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-17 returned error [No such file or directory] >[2018-08-18 11:35:03.751213] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751220] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751251] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751262] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751270] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-18 returned error [No such file or directory] >[2018-08-18 11:35:03.751359] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751380] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751384] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751406] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751415] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-19 returned error [No such file or directory] >[2018-08-18 11:35:03.751432] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751457] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751479] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751506] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751523] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-20 returned error [No such file or directory] >[2018-08-18 11:35:03.751530] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751561] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751617] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751641] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751652] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-21 returned error [No such file or directory] >[2018-08-18 11:35:03.751695] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751734] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751775] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9000e660, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751773] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751804] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751821] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-22 returned error [No such file or directory] >[2018-08-18 11:35:03.751860] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751920] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c001f40, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751932] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c001f40, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751931] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751978] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.751995] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.752007] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-23 returned error [No such file or directory] >[2018-08-18 11:35:03.752043] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.752078] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-9: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.752080] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.752090] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9000e660, vol0-client-9 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.752105] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-11: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.752109] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-10: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.752112] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.752121] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9000e660, vol0-client-10 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.752116] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9000e660, vol0-client-11 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.752134] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.752148] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-24 returned error [No such file or directory] >[2018-08-18 11:35:03.752205] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.752256] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.752257] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.752289] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.752305] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-25 returned error [No such file or directory] >[2018-08-18 11:35:03.752355] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.752425] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.752448] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-27: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.752456] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c001f40, vol0-client-27 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.752461] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.752503] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-29: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.752498] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-28: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.752499] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.752520] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c001f40, vol0-client-28 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.752521] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c001f40, vol0-client-29 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.752534] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-26 returned error [No such file or directory] >[2018-08-18 11:35:03.752553] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.752614] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.752741] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.752814] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.752815] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.752847] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.752863] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-28 returned error [No such file or directory] >[2018-08-18 11:35:03.753025] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.753040] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.753048] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9800aea0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.753166] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9800aea0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.753176] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.753214] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.753207] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.753246] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.753263] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-30 returned error [No such file or directory] >[2018-08-18 11:35:03.753305] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.753377] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.753427] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-81: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.753436] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-82: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.753456] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9800aea0, vol0-client-82 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.753460] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9800aea0, vol0-client-81 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.753462] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-83: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.753485] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9800aea0, vol0-client-83 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.753573] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.753627] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.753629] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.753650] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.753658] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-32 returned error [No such file or directory] >[2018-08-18 11:35:03.753724] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.753788] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.753793] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.753808] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.753817] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-33 returned error [No such file or directory] >[2018-08-18 11:35:03.754074] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.754096] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.754112] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.754134] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.754150] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on vol0-replicate-35 returned error [No such file or directory] >[2018-08-18 11:35:03.754730] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 11:35:03.756266] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9000cae0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.756272] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c009700, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.756272] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9000cae0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.756312] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c009700, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.756523] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-87: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.756539] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-88: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.756546] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9000cae0, vol0-client-87 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.756566] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9000cae0, vol0-client-88 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.756567] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-89: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.756593] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9000cae0, vol0-client-89 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.756643] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-93: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.756655] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c009700, vol0-client-93 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.756668] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-94: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.756675] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-95: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.756679] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c009700, vol0-client-94 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.756690] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c009700, vol0-client-95 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.758045] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-27: lk op is for a transaction >[2018-08-18 11:35:03.761931] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 11:35:03.762279] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 11:35:03.765287] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 11:35:03.766317] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:03.767291] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-30: lk op is for a transaction >[2018-08-18 11:35:03.767881] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 11:35:03.767893] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 11:35:03.768759] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:03.769991] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 11:35:03.770349] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 11:35:03.771677] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 11:35:03.771710] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 11:35:03.771832] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 11:35:03.772276] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-vol0-dht: Found anomalies in /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m (gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0). Holes=1 overlaps=0 >[2018-08-18 11:35:03.772301] D [MSGID: 0] [dht-common.c:943:dht_lookup_dir_cbk] 0-vol0-dht: fixing assignment on /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m >[2018-08-18 11:35:03.772350] D [MSGID: 0] [dht-selfheal.c:1879:dht_selfheal_layout_new_directory] 0-vol0-dht: chunk size = 0xffffffff / 274638528 = 15.638619 >[2018-08-18 11:35:03.772366] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-26 >[2018-08-18 11:35:03.772383] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-27 >[2018-08-18 11:35:03.772393] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-28 >[2018-08-18 11:35:03.772402] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-29 >[2018-08-18 11:35:03.772411] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-3 >[2018-08-18 11:35:03.772421] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-30 >[2018-08-18 11:35:03.772430] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-31 >[2018-08-18 11:35:03.772439] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-32 >[2018-08-18 11:35:03.772449] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-33 >[2018-08-18 11:35:03.772458] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-34 >[2018-08-18 11:35:03.772467] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-35 >[2018-08-18 11:35:03.772476] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-4 >[2018-08-18 11:35:03.772486] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-5 >[2018-08-18 11:35:03.772495] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 11:35:03.772495] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-6 >[2018-08-18 11:35:03.772524] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-7 >[2018-08-18 11:35:03.772534] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-8 >[2018-08-18 11:35:03.772543] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-9 >[2018-08-18 11:35:03.772552] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-0 >[2018-08-18 11:35:03.772562] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-1 >[2018-08-18 11:35:03.772571] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-10 >[2018-08-18 11:35:03.772580] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-11 >[2018-08-18 11:35:03.772590] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-12 >[2018-08-18 11:35:03.772606] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-13 >[2018-08-18 11:35:03.772619] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-14 >[2018-08-18 11:35:03.772628] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-15 >[2018-08-18 11:35:03.772637] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-16 >[2018-08-18 11:35:03.772650] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-17 >[2018-08-18 11:35:03.772660] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-18 >[2018-08-18 11:35:03.772669] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-19 >[2018-08-18 11:35:03.772678] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-2 >[2018-08-18 11:35:03.772687] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-20 >[2018-08-18 11:35:03.772696] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-21 >[2018-08-18 11:35:03.772705] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-22 >[2018-08-18 11:35:03.772715] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-23 >[2018-08-18 11:35:03.772724] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-24 >[2018-08-18 11:35:03.772734] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-25 >[2018-08-18 11:35:03.773211] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-22: lk op is for a transaction >[2018-08-18 11:35:03.773242] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 11:35:03.773662] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 11:35:03.773774] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 11:35:03.773871] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:03.774389] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-20: lk op is for a transaction >[2018-08-18 11:35:03.774581] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 11:35:03.774756] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 11:35:03.774866] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-0: remote operation failed [Stale file handle] >[2018-08-18 11:35:03.774882] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbf98027e10, vol0-client-0 returned -1 error: Stale file handle [Stale file handle] >[2018-08-18 11:35:03.775026] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 11:35:03.775527] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 11:35:03.776195] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 11:35:03.776437] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-26: lk op is for a transaction >[2018-08-18 11:35:03.776578] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 11:35:03.776595] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-1: remote operation failed [Stale file handle] >[2018-08-18 11:35:03.776621] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbf98027e10, vol0-client-1 returned -1 error: Stale file handle [Stale file handle] >[2018-08-18 11:35:03.776688] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 11:35:03.776767] D [MSGID: 0] [afr-common.c:3592:afr_fop_lock_unwind] 0-stack-trace: stack-address: 0x7fbf98027e10, vol0-replicate-0 returned -1 error: Stale file handle [Stale file handle] >[2018-08-18 11:35:03.778266] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:03.778834] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-23: lk op is for a transaction >[2018-08-18 11:35:03.780033] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-54: remote operation failed [Stale file handle] >[2018-08-18 11:35:03.780051] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbf98027e10, vol0-client-54 returned -1 error: Stale file handle [Stale file handle] >[2018-08-18 11:35:03.780162] D [MSGID: 0] [dht-selfheal.c:1879:dht_selfheal_layout_new_directory] 0-vol0-dht: chunk size = 0xffffffff / 274638528 = 15.638619 >[2018-08-18 11:35:03.780176] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-26 >[2018-08-18 11:35:03.780183] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-27 >[2018-08-18 11:35:03.780189] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-28 >[2018-08-18 11:35:03.780195] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-29 >[2018-08-18 11:35:03.780200] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-3 >[2018-08-18 11:35:03.780206] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-30 >[2018-08-18 11:35:03.780211] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-31 >[2018-08-18 11:35:03.780216] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-32 >[2018-08-18 11:35:03.780222] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-33 >[2018-08-18 11:35:03.780227] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-34 >[2018-08-18 11:35:03.780232] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-35 >[2018-08-18 11:35:03.780237] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-4 >[2018-08-18 11:35:03.780242] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-5 >[2018-08-18 11:35:03.780248] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-6 >[2018-08-18 11:35:03.780253] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-7 >[2018-08-18 11:35:03.780258] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-8 >[2018-08-18 11:35:03.780263] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-9 >[2018-08-18 11:35:03.780268] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-0 >[2018-08-18 11:35:03.780274] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-1 >[2018-08-18 11:35:03.780286] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-10 >[2018-08-18 11:35:03.780291] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-11 >[2018-08-18 11:35:03.780296] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-12 >[2018-08-18 11:35:03.780301] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-13 >[2018-08-18 11:35:03.780306] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-14 >[2018-08-18 11:35:03.780311] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-15 >[2018-08-18 11:35:03.780316] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-16 >[2018-08-18 11:35:03.780321] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-17 >[2018-08-18 11:35:03.780326] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-18 >[2018-08-18 11:35:03.780331] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-19 >[2018-08-18 11:35:03.780336] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-2 >[2018-08-18 11:35:03.780341] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-20 >[2018-08-18 11:35:03.780346] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-21 >[2018-08-18 11:35:03.780351] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-22 >[2018-08-18 11:35:03.780356] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-23 >[2018-08-18 11:35:03.780361] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-24 >[2018-08-18 11:35:03.780366] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-25 >[2018-08-18 11:35:03.780422] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 11:35:03.782821] D [MSGID: 109036] [dht-common.c:9587:dht_log_new_layout_for_dir_selfheal] 0-vol0-dht: Setting layout of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m with [Subvol_name: vol0-replicate-0, Err: -1 , Start: 2028178999 , Stop: 2147483645 , Hash: 1 ], [Subvol_name: vol0-replicate-1, Err: -1 , Start: 2147483646 , Stop: 2266788292 , Hash: 1 ], [Subvol_name: vol0-replicate-10, Err: -1 , Start: 2266788293 , Stop: 2386092939 , Hash: 1 ], [Subvol_name: vol0-replicate-11, Err: -1 , Start: 2386092940 , Stop: 2505397586 , Hash: 1 ], [Subvol_name: vol0-replicate-12, Err: -1 , Start: 2505397587 , Stop: 2624702233 , Hash: 1 ], [Subvol_name: vol0-replicate-13, Err: -1 , Start: 2624702234 , Stop: 2744006880 , Hash: 1 ], [Subvol_name: vol0-replicate-14, Err: -1 , Start: 2744006881 , Stop: 2863311527 , Hash: 1 ], [Subvol_name: vol0-replicate-15, Err: -1 , Start: 2863311528 , Stop: 2982616174 , Hash: 1 ], [Subvol_name: vol0-replicate-16, Err: -1 , Start: 2982616175 , Stop: 3101920821 , Hash: 1 ], [Subvol_name: vol0-replicate-17, Err: -1 , Start: 3101920822 , Stop: 3221225468 , Hash: 1 ], [Subvol_name: vol0-replicate-18, Err: -1 , Start: 3221225469 , Stop: 3340530115 , Hash: 1 ], [Subvol_name: vol0-replicate-19, Err: -1 , Start: 3340530116 , Stop: 3459834762 , Hash: 1 ], [Subvol_name: vol0-replicate-2, Err: -1 , Start: 3459834763 , Stop: 3579139409 , Hash: 1 ], [Subvol_name: vol0-replicate-20, Err: -1 , Start: 3579139410 , Stop: 3698444056 , Hash: 1 ], [Subvol_name: vol0-replicate-21, Err: -1 , Start: 3698444057 , Stop: 3817748703 , Hash: 1 ], [Subvol_name: vol0-replicate-22, Err: -1 , Start: 3817748704 , Stop: 3937053350 , Hash: 1 ], [Subvol_name: vol0-replicate-23, Err: -1 , Start: 3937053351 , Stop: 4056357997 , Hash: 1 ], [Subvol_name: vol0-replicate-24, Err: -1 , Start: 4056357998 , Stop: 4175662644 , Hash: 1 ], [Subvol_name: vol0-replicate-25, Err: -1 , Start: 4175662645 , Stop: 4294967295 , Hash: 1 ], [Subvol_name: vol0-replicate-26, Err: -1 , Start: 0 , Stop: 119304646 , Hash: 1 ], [Subvol_name: vol0-replicate-27, Err: -1 , Start: 119304647 , Stop: 238609293 , Hash: 1 ], [Subvol_name: vol0-replicate-28, Err: -1 , Start: 238609294 , Stop: 357913940 , Hash: 1 ], [Subvol_name: vol0-replicate-29, Err: -1 , Start: 357913941 , Stop: 477218587 , Hash: 1 ], [Subvol_name: vol0-replicate-3, Err: -1 , Start: 477218588 , Stop: 596523234 , Hash: 1 ], [Subvol_name: vol0-replicate-30, Err: -1 , Start: 596523235 , Stop: 715827881 , Hash: 1 ], [Subvol_name: vol0-replicate-31, Err: -1 , Start: 715827882 , Stop: 835132528 , Hash: 1 ], [Subvol_name: vol0-replicate-32, Err: -1 , Start: 835132529 , Stop: 954437175 , Hash: 1 ], [Subvol_name: vol0-replicate-33, Err: -1 , Start: 954437176 , Stop: 1073741822 , Hash: 1 ], [Subvol_name: vol0-replicate-34, Err: -1 , Start: 1073741823 , Stop: 1193046469 , Hash: 1 ], [Subvol_name: vol0-replicate-35, Err: -1 , Start: 1193046470 , Stop: 1312351116 , Hash: 1 ], [Subvol_name: vol0-replicate-4, Err: -1 , Start: 1312351117 , Stop: 1431655763 , Hash: 1 ], [Subvol_name: vol0-replicate-5, Err: -1 , Start: 1431655764 , Stop: 1550960410 , Hash: 1 ], [Subvol_name: vol0-replicate-6, Err: -1 , Start: 1550960411 , Stop: 1670265057 , Hash: 1 ], [Subvol_name: vol0-replicate-7, Err: -1 , Start: 1670265058 , Stop: 1789569704 , Hash: 1 ], [Subvol_name: vol0-replicate-8, Err: -1 , Start: 1789569705 , Stop: 1908874351 , Hash: 1 ], [Subvol_name: vol0-replicate-9, Err: -1 , Start: 1908874352 , Stop: 2028178998 , Hash: 1 ], >[2018-08-18 11:35:03.783047] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-0: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.783109] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-1: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.783166] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.783284] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 11:35:03.783285] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.783339] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 11:35:03.783355] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:03.783397] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.783393] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-12: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.783472] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.783513] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:03.783554] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-15: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.783611] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 11:35:03.783663] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.783670] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 11:35:03.783693] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 11:35:03.783741] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-17: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.783782] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 11:35:03.783841] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-18: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.783868] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 11:35:03.783954] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 11:35:03.783992] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-19: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.784060] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-2: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.784064] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 11:35:03.784169] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-20: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.784186] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 11:35:03.784224] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.784280] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 11:35:03.784315] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-22: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.784378] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-20: lk op is for a transaction >[2018-08-18 11:35:03.784392] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-23: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.784429] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 11:35:03.784487] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.784507] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-22: lk op is for a transaction >[2018-08-18 11:35:03.784566] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-25: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.784582] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-23: lk op is for a transaction >[2018-08-18 11:35:03.784618] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-26: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.784680] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:03.784716] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-27: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.784762] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 11:35:03.784787] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-28: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.784857] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-26: lk op is for a transaction >[2018-08-18 11:35:03.784898] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-29: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.785001] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-3: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.785010] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 11:35:03.785032] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-27: lk op is for a transaction >[2018-08-18 11:35:03.785076] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-30: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.785121] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 11:35:03.785135] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.785217] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 11:35:03.785223] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-32: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.785311] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-33: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.785328] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-30: lk op is for a transaction >[2018-08-18 11:35:03.785370] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 11:35:03.785404] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-34: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.785453] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 11:35:03.785521] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 11:35:03.785560] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.785593] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 11:35:03.785637] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-5: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.785729] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-6: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.785799] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:03.785826] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.785838] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 11:35:03.785874] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-8: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.785925] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-35: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.785951] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-9: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.785964] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 11:35:03.786060] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 11:35:03.786188] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 11:35:03.786267] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 11:35:03.786340] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 11:35:03.787088] D [MSGID: 0] [dht-helper.c:1105:dht_subvol_get_hashed] 0-vol0-dht: Missing layout. path=/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88, parent gfid =3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 >[2018-08-18 11:35:03.787121] D [MSGID: 0] [dht-common.c:2716:dht_lookup] 0-vol0-dht: no subvolume in layout for path=/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88, checking on all the subvols to see if it is a directory >[2018-08-18 11:35:03.787133] D [MSGID: 0] [dht-common.c:2730:dht_lookup] 0-vol0-dht: Found null hashed subvol. Calling lookup on all nodes. >[2018-08-18 11:35:03.787451] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787474] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787517] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787543] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787591] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787591] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787618] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787624] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787631] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787631] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-0 returned error [No such file or directory] >[2018-08-18 11:35:03.787657] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-1 returned error [No such file or directory] >[2018-08-18 11:35:03.787694] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787700] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787714] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787733] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787737] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787752] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787760] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-2 returned error [No such file or directory] >[2018-08-18 11:35:03.787771] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787778] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787788] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787807] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-3 returned error [No such file or directory] >[2018-08-18 11:35:03.787812] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787829] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787838] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-4 returned error [No such file or directory] >[2018-08-18 11:35:03.787903] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787906] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787961] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787977] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.787992] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-5 returned error [No such file or directory] >[2018-08-18 11:35:03.787995] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788039] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788077] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788096] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788105] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-6 returned error [No such file or directory] >[2018-08-18 11:35:03.788124] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788159] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788172] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788193] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788206] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-7 returned error [No such file or directory] >[2018-08-18 11:35:03.788241] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788256] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788265] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788282] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788291] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-8 returned error [No such file or directory] >[2018-08-18 11:35:03.788332] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788403] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788421] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788465] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788481] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788500] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788510] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-10 returned error [No such file or directory] >[2018-08-18 11:35:03.788536] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788571] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788576] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788597] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788608] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788626] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-11 returned error [No such file or directory] >[2018-08-18 11:35:03.788668] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788691] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788697] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788741] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788743] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788743] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788783] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-12 returned error [No such file or directory] >[2018-08-18 11:35:03.788784] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788790] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788806] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-13 returned error [No such file or directory] >[2018-08-18 11:35:03.788849] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788893] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788922] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788959] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788954] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-14 returned error [No such file or directory] >[2018-08-18 11:35:03.788962] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.788952] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789001] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789016] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-15 returned error [No such file or directory] >[2018-08-18 11:35:03.789043] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789087] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789107] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789129] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789152] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-16 returned error [No such file or directory] >[2018-08-18 11:35:03.789171] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789195] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789216] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789239] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789254] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-17 returned error [No such file or directory] >[2018-08-18 11:35:03.789357] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789358] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789451] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789459] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789516] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789541] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789544] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789548] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789552] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789553] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789569] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789582] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789593] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789595] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-18 returned error [No such file or directory] >[2018-08-18 11:35:03.789592] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789627] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-20 returned error [No such file or directory] >[2018-08-18 11:35:03.789626] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789637] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789671] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-9 returned error [No such file or directory] >[2018-08-18 11:35:03.789691] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-19 returned error [No such file or directory] >[2018-08-18 11:35:03.789706] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789723] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789730] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789738] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789738] D [MSGID: 0] [dht-selfheal.c:1206:dht_selfheal_dir_mkdir_setacl] 0-vol0-dht: ACL_DEFAULT xattr not present >[2018-08-18 11:35:03.789745] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789758] D [MSGID: 0] [dht-selfheal.c:1219:dht_selfheal_dir_mkdir_setacl] 0-vol0-dht: ACL_ACCESS xattr not present >[2018-08-18 11:35:03.789774] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-0 >[2018-08-18 11:35:03.789779] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789779] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-21 returned error [No such file or directory] >[2018-08-18 11:35:03.789780] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789788] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789819] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789846] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-22 returned error [No such file or directory] >[2018-08-18 11:35:03.789854] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789863] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-23 returned error [No such file or directory] >[2018-08-18 11:35:03.789913] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-1 >[2018-08-18 11:35:03.789915] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789963] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789969] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789983] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.789992] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-24 returned error [No such file or directory] >[2018-08-18 11:35:03.790003] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-10 >[2018-08-18 11:35:03.790009] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-0: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.790037] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790071] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-11 >[2018-08-18 11:35:03.790077] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790110] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-1: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.790119] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-0: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.790109] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790133] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8057cc0, vol0-client-0 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.790133] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-1: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.790146] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790150] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-12 >[2018-08-18 11:35:03.790150] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8057cc0, vol0-client-1 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.790150] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790166] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-2: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.790170] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-25 returned error [No such file or directory] >[2018-08-18 11:35:03.790177] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8057cc0, vol0-client-2 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.790188] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790191] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-10: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.790201] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 11:35:03.790216] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790221] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-13 >[2018-08-18 11:35:03.790241] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790245] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790252] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-4: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.790274] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-3: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.790274] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-26 returned error [No such file or directory] >[2018-08-18 11:35:03.790287] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-11: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.790282] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-5: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.790309] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8059d00, vol0-client-5 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.790281] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8059d00, vol0-client-4 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.790311] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-14 >[2018-08-18 11:35:03.790292] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8059d00, vol0-client-3 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.790335] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790337] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 11:35:03.790350] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790368] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-30: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.790370] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790383] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-12: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.790396] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8004ed0, vol0-client-30 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.790397] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-15 >[2018-08-18 11:35:03.790417] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-31: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.790422] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790430] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-32: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.790417] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-27 returned error [No such file or directory] >[2018-08-18 11:35:03.790457] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8004ed0, vol0-client-32 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.790471] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8004ed0, vol0-client-31 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.790470] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790486] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-0 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.790481] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-17 >[2018-08-18 11:35:03.790496] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:03.790512] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-34: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.790510] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-33: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.790529] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8032900, vol0-client-34 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.790520] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790519] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.790554] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790554] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8032900, vol0-client-33 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.790566] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-28 returned error [No such file or directory] >[2018-08-18 11:35:03.790577] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-18 >[2018-08-18 11:35:03.790578] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-35: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.790616] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790607] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-13: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.790628] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8032900, vol0-client-35 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.790626] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-14: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.790625] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790652] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:03.790667] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-36: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.790669] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-19 >[2018-08-18 11:35:03.790678] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-1 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.790692] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd805ad20, vol0-client-36 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.790695] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.790708] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790725] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-37: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.790734] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790742] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790729] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-38: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.790742] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd805ad20, vol0-client-37 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.790730] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790772] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd805ad20, vol0-client-38 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.790780] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-29 returned error [No such file or directory] >[2018-08-18 11:35:03.790781] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790783] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-2 >[2018-08-18 11:35:03.790817] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790816] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 11:35:03.790841] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-15: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.790854] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790865] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790874] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-30 returned error [No such file or directory] >[2018-08-18 11:35:03.790883] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-10 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.790902] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.790906] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790912] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-20 >[2018-08-18 11:35:03.790925] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790944] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-31 returned error [No such file or directory] >[2018-08-18 11:35:03.790954] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-42: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.790944] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790968] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8056490, vol0-client-42 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.790946] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-17: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.790975] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.790991] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-39: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791015] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-18: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.791021] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8054c60, vol0-client-39 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791017] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-44: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791022] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-21 >[2018-08-18 11:35:03.791043] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.791044] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-43: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791055] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.791045] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8056490, vol0-client-44 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791071] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-32 returned error [No such file or directory] >[2018-08-18 11:35:03.791067] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8056490, vol0-client-43 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791070] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-40: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791099] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 11:35:03.791100] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-22 >[2018-08-18 11:35:03.791104] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-41: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791102] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8054c60, vol0-client-40 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791126] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8054c60, vol0-client-41 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791144] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-11 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791141] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.791156] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 11:35:03.791162] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.791169] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.791173] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-23 >[2018-08-18 11:35:03.791186] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-19: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.791204] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.791209] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.791206] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.791251] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.791256] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-24 >[2018-08-18 11:35:03.791252] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-45: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791267] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-2: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.791272] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.791268] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-46: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791281] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-33 returned error [No such file or directory] >[2018-08-18 11:35:03.791280] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8055470, vol0-client-45 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791316] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8055470, vol0-client-46 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791315] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.791326] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-12 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791340] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.791342] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-25 >[2018-08-18 11:35:03.791345] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-34 returned error [No such file or directory] >[2018-08-18 11:35:03.791329] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-47: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791346] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.791367] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8055470, vol0-client-47 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791368] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-20: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.791373] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.791386] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-52: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791386] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 11:35:03.791402] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8053430, vol0-client-52 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791413] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-53: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791410] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-51: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791434] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-26 >[2018-08-18 11:35:03.791440] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-55: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791463] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8057cc0, vol0-client-55 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791437] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8053430, vol0-client-53 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791472] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8053430, vol0-client-51 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791466] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-54: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791476] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-56: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791491] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8057cc0, vol0-client-54 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791492] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 11:35:03.791511] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8057cc0, vol0-client-56 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791508] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-28 >[2018-08-18 11:35:03.791508] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-21: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.791522] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-14 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791539] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-22: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.791543] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 11:35:03.791556] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.791584] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-57: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791586] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-13 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791595] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd805a510, vol0-client-57 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791623] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-30 >[2018-08-18 11:35:03.791623] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.791623] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-58: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791642] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-59: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791661] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-7: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791679] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-8: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791671] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd805a510, vol0-client-58 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791688] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8059d00, vol0-client-7 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791659] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-23: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.791693] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8059d00, vol0-client-8 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791691] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd805a510, vol0-client-59 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791703] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-32 >[2018-08-18 11:35:03.791713] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-6: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791721] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 11:35:03.791728] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8059d00, vol0-client-6 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791732] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-24: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.791735] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-62: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791749] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-61: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791751] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 11:35:03.791751] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8004ed0, vol0-client-62 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791769] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8004ed0, vol0-client-61 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791786] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-33 >[2018-08-18 11:35:03.791790] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-60: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791804] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-25: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.791817] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8004ed0, vol0-client-60 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791820] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-15 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791835] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-20: lk op is for a transaction >[2018-08-18 11:35:03.791856] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-63: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791862] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-26: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.791866] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.791882] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8055c80, vol0-client-63 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791886] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-66: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791896] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-35 >[2018-08-18 11:35:03.791919] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-64: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791935] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8055c80, vol0-client-64 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791918] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd80093b0, vol0-client-66 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791937] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-65: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791939] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-17 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791953] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-67: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791955] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8055c80, vol0-client-65 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791962] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd80093b0, vol0-client-67 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791965] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-68: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.791984] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 11:35:03.791992] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-28: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.791994] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd80093b0, vol0-client-68 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.791965] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.792008] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-4 >[2018-08-18 11:35:03.792014] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-22: lk op is for a transaction >[2018-08-18 11:35:03.792002] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-18 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792035] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.792053] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-69: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792060] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-70: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792063] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-71: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792073] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8032900, vol0-client-70 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792071] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8032900, vol0-client-69 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792086] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-5 >[2018-08-18 11:35:03.792090] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8032900, vol0-client-71 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792111] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-72: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792112] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-23: lk op is for a transaction >[2018-08-18 11:35:03.792114] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-30: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.792133] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-32: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.792137] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-73: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792135] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd80584d0, vol0-client-72 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792165] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd80584d0, vol0-client-73 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792173] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-19 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792167] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-6 >[2018-08-18 11:35:03.792198] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.792201] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-74: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792206] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-77: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792222] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-76: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792224] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-2 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792229] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8002100, vol0-client-77 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792220] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd80584d0, vol0-client-74 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792235] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8002100, vol0-client-76 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792254] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.792272] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-78: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792278] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m on subvol vol0-replicate-8 >[2018-08-18 11:35:03.792280] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-33: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.792285] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd805ad20, vol0-client-78 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792280] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-75: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792280] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:03.792323] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-79: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792331] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-20 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792344] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd805ad20, vol0-client-79 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792346] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.792352] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8002100, vol0-client-75 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792350] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-80: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792376] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd805ad20, vol0-client-80 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792386] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-84: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792377] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 11:35:03.792400] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-21 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792390] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-26: lk op is for a transaction >[2018-08-18 11:35:03.792431] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-85: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792436] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-86: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792449] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8056ca0, vol0-client-84 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792455] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.792456] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8056ca0, vol0-client-86 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792464] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8056ca0, vol0-client-85 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792474] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-22 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792495] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.792497] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-4: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.792504] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 11:35:03.792500] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-90: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792521] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-5: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.792529] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8056490, vol0-client-90 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792522] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-98: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792543] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-96: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792551] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8054c60, vol0-client-98 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792556] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8054c60, vol0-client-96 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792568] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-92: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792578] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8056490, vol0-client-92 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792582] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-97: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792592] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-23 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792609] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8054c60, vol0-client-97 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792603] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-91: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792606] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-6: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.792627] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 11:35:03.792628] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8056490, vol0-client-91 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792659] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-99: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792663] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.792666] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.792672] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-100: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792701] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-30: lk op is for a transaction >[2018-08-18 11:35:03.792686] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd803de60, vol0-client-99 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792725] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd803de60, vol0-client-100 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792722] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-101: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792731] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.792746] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-8: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.792748] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd803de60, vol0-client-101 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792749] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-35 returned error [No such file or directory] >[2018-08-18 11:35:03.792750] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-24 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792768] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 11:35:03.792777] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.792777] D [MSGID: 0] [dht-common.c:964:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.792784] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-26 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792786] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-25 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792816] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-13: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792818] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.792814] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-12: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792827] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8053430, vol0-client-13 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792857] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-15: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792822] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.792858] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8053430, vol0-client-12 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792861] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.792866] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-14: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792886] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-16: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792891] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8057cc0, vol0-client-15 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792898] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8053430, vol0-client-14 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792902] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.792911] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-17: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792906] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-28 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792916] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:03.792921] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-18: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792928] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8057cc0, vol0-client-17 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792931] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.792921] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.792921] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8057cc0, vol0-client-16 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792958] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-20: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792960] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.792964] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-19: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.792965] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 11:35:03.792982] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd803d650, vol0-client-19 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792958] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd803d650, vol0-client-18 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.792985] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd803d650, vol0-client-20 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.793009] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-32 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.793010] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 11:35:03.793017] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-24: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.793031] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-25: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.793019] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-30 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.793039] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8059d00, vol0-client-24 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.793036] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.793041] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-26: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.793066] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.793069] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8059d00, vol0-client-25 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.793080] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-33 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.793080] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8059d00, vol0-client-26 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.793111] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.793116] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-35: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.793118] D [MSGID: 0] [dht-helper.c:1105:dht_subvol_get_hashed] 0-vol0-dht: Missing layout. path=/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88, parent gfid =3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 >[2018-08-18 11:35:03.793128] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-4 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.793120] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 11:35:03.793146] E [MSGID: 109011] [dht-common.c:7248:dht_create] 0-vol0-dht: no subvolume in layout for path=/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 >[2018-08-18 11:35:03.793154] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-5 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.793155] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.793165] D [MSGID: 0] [dht-common.c:7331:dht_create] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-dht returned -1 error: Input/output error [Input/output error] >[2018-08-18 11:35:03.793179] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-6 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.793187] D [MSGID: 0] [md-cache.c:1703:mdc_create_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-md-cache returned -1 error: Input/output error [Input/output error] >[2018-08-18 11:35:03.793189] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.793206] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 11:35:03.793198] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.793205] D [MSGID: 0] [io-stats.c:1941:io_stats_create_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0 returned -1 error: Input/output error [Input/output error] >[2018-08-18 11:35:03.793240] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-105: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.793253] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-106: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.793260] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8055470, vol0-client-105 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.793267] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8055470, vol0-client-106 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.793268] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-107: remote operation failed. Path: /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m [File exists] >[2018-08-18 11:35:03.793289] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8055470, vol0-client-107 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.793289] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-8 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.793303] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 11:35:03.793315] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.793453] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-35 returned -1 error: File exists [File exists] >[2018-08-18 11:35:03.793465] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m, gfid = 3d217ae6-2cc8-4fa2-b9fc-243624a65dc0 [File exists] >[2018-08-18 11:35:03.793961] E [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-54: remote operation failed [Invalid argument] >[2018-08-18 11:35:03.793978] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd404ff80, vol0-client-54 returned -1 error: Invalid argument [Invalid argument] >[2018-08-18 11:35:03.795170] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-0: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.795173] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-1: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.795292] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.795289] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.795335] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-12: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.795412] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 11:35:03.795410] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 11:35:03.795410] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.795516] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:03.795518] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.795538] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:03.795542] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 11:35:03.795545] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.795642] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-15: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.795657] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 11:35:03.795716] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 11:35:03.795765] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-17: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.795819] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 11:35:03.795860] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 11:35:03.795880] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-18: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.795941] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-19: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.795991] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 11:35:03.796042] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-2: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.796110] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 11:35:03.796143] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-20: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.796140] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 11:35:03.796213] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 11:35:03.796246] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.796338] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-20: lk op is for a transaction >[2018-08-18 11:35:03.796390] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-22: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.796460] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 11:35:03.796466] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-23: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.796531] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-27: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.796584] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-22: lk op is for a transaction >[2018-08-18 11:35:03.796616] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.796665] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-25: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.796694] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-23: lk op is for a transaction >[2018-08-18 11:35:03.796708] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-3: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.796735] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-29: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.796780] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-27: lk op is for a transaction >[2018-08-18 11:35:03.796783] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-26: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.796813] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.796836] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:03.796903] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 11:35:03.796975] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 11:35:03.796993] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-30: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.797000] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-28: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.797040] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-26: lk op is for a transaction >[2018-08-18 11:35:03.797064] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 11:35:03.797060] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-34: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.797070] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 11:35:03.797152] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-32: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.797178] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-30: lk op is for a transaction >[2018-08-18 11:35:03.797199] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 11:35:03.797229] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-33: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.797322] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 11:35:03.797382] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 11:35:03.797434] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 11:35:03.797476] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.797496] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.797515] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-5: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.797623] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-9: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.797687] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-6: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.797721] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 11:35:03.797737] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:03.797760] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 11:35:03.797766] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-35: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.797802] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-8: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:03.797853] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 11:35:03.797904] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 11:35:03.797966] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 11:35:03.798081] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 11:35:03.798167] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-0: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.798181] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfc0077770, vol0-client-0 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.798196] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-2: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.798204] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfc0077770, vol0-client-2 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.798204] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-1: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.798224] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfc0077770, vol0-client-1 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.904382] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88 on vol0-replicate-4 >[2018-08-18 11:35:03.904629] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.904652] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.904673] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.904688] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.904696] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88 with op_ret -1 [No such file or directory] >[2018-08-18 11:35:03.904705] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88 missing on subvol vol0-replicate-4 >[2018-08-18 11:35:03.904712] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:03.904914] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.904923] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.904947] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.904956] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.904976] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.904978] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905000] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905005] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-0 >[2018-08-18 11:35:03.905006] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905025] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905030] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905043] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-1 >[2018-08-18 11:35:03.905041] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905063] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905069] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905080] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-2 >[2018-08-18 11:35:03.905083] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905110] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905140] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905142] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905145] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905181] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-3 >[2018-08-18 11:35:03.905207] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905213] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905210] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905248] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905269] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905286] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905278] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-5 >[2018-08-18 11:35:03.905290] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905308] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905327] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905340] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905333] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905353] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-6 >[2018-08-18 11:35:03.905375] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905374] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905372] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905406] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905415] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-7 >[2018-08-18 11:35:03.905420] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905419] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905431] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-8 >[2018-08-18 11:35:03.905463] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905469] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905489] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905501] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-9 >[2018-08-18 11:35:03.905507] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905526] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905557] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905575] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905579] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905588] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-10 >[2018-08-18 11:35:03.905621] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905622] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905656] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905658] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905676] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-11 >[2018-08-18 11:35:03.905677] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905709] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905738] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905749] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905749] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905750] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905777] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905780] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905782] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-12 >[2018-08-18 11:35:03.905796] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-13 >[2018-08-18 11:35:03.905800] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905806] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905829] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905842] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905842] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-14 >[2018-08-18 11:35:03.905852] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905879] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905894] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905902] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-15 >[2018-08-18 11:35:03.905925] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905936] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905934] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.905982] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906001] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-16 >[2018-08-18 11:35:03.906022] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906019] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906024] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906048] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906056] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-17 >[2018-08-18 11:35:03.906068] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906091] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906098] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906131] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906138] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-18 >[2018-08-18 11:35:03.906154] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906162] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906198] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906211] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906211] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906221] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906240] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906242] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906251] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-19 >[2018-08-18 11:35:03.906269] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906265] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906276] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906263] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-20 >[2018-08-18 11:35:03.906300] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906305] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906307] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906328] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-21 >[2018-08-18 11:35:03.906334] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906351] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906362] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-22 >[2018-08-18 11:35:03.906364] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906401] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906414] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906416] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906436] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906442] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906451] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-23 >[2018-08-18 11:35:03.906458] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906473] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906481] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-24 >[2018-08-18 11:35:03.906503] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906504] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906540] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906559] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906567] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906574] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906590] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-25 >[2018-08-18 11:35:03.906605] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906627] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906631] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906633] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906647] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906659] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-26 >[2018-08-18 11:35:03.906658] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906684] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-27 >[2018-08-18 11:35:03.906689] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906689] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906728] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906716] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906758] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906763] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906759] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906789] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906790] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-28 >[2018-08-18 11:35:03.906797] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-29 >[2018-08-18 11:35:03.906824] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906840] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906840] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906865] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906866] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906874] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906896] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-30 >[2018-08-18 11:35:03.906900] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906925] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906928] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906945] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-31 >[2018-08-18 11:35:03.906951] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906978] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.906995] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.907008] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-32 >[2018-08-18 11:35:03.907015] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.907026] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.907027] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.907053] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.907068] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.907068] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.907085] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.907088] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-4 >[2018-08-18 11:35:03.907093] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.907097] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.907106] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-33 >[2018-08-18 11:35:03.907116] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.907129] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-34 >[2018-08-18 11:35:03.907134] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.907177] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.907181] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.907205] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.907219] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-35 >[2018-08-18 11:35:03.907235] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-4 cached_subvol null >[2018-08-18 11:35:03.907247] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88 >[2018-08-18 11:35:03.907260] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.907274] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.907290] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.907305] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.907319] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa806f660, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.907451] D [MSGID: 0] [dht-common.c:6923:dht_create_wind_to_avail_subvol] 0-vol0-dht: creating /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88 on vol0-replicate-4 >[2018-08-18 11:35:03.907638] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-4: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:03.924253] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m on vol0-replicate-4 >[2018-08-18 11:35:03.924685] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49192: ping timer event already removed >[2018-08-18 11:35:03.924969] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49192: ping timer event already removed >[2018-08-18 11:35:03.925051] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.925234] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49192: ping timer event already removed >[2018-08-18 11:35:03.925305] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.925755] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c015830, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.925764] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c015830, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.926067] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-12: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.926091] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf8c015830, vol0-client-12 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.926095] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-13: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.926096] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-14: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.926117] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf8c015830, vol0-client-13 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.926122] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf8c015830, vol0-client-14 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.933867] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m with op_ret 0 >[2018-08-18 11:35:03.934170] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49188: ping timer event already removed >[2018-08-18 11:35:03.934453] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49188: ping timer event already removed >[2018-08-18 11:35:03.934755] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49188: ping timer event already removed >[2018-08-18 11:35:03.935099] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49189: ping timer event already removed >[2018-08-18 11:35:03.935168] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.935367] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49189: ping timer event already removed >[2018-08-18 11:35:03.935454] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.935662] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49189: ping timer event already removed >[2018-08-18 11:35:03.935735] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.935770] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.935785] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m on vol0-replicate-1 returned error [No such file or directory] >[2018-08-18 11:35:03.935981] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49190: ping timer event already removed >[2018-08-18 11:35:03.936069] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.936260] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49190: ping timer event already removed >[2018-08-18 11:35:03.936343] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.936539] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49190: ping timer event already removed >[2018-08-18 11:35:03.936580] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.936610] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.936632] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m on vol0-replicate-2 returned error [No such file or directory] >[2018-08-18 11:35:03.936849] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49191: ping timer event already removed >[2018-08-18 11:35:03.936933] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.937128] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49191: ping timer event already removed >[2018-08-18 11:35:03.937217] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.937405] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49191: ping timer event already removed >[2018-08-18 11:35:03.937474] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.937516] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.937546] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m on vol0-replicate-3 returned error [No such file or directory] >[2018-08-18 11:35:03.937854] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49193: ping timer event already removed >[2018-08-18 11:35:03.938119] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49193: ping timer event already removed >[2018-08-18 11:35:03.938365] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49193: ping timer event already removed >[2018-08-18 11:35:03.938699] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49194: ping timer event already removed >[2018-08-18 11:35:03.938763] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.938950] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49194: ping timer event already removed >[2018-08-18 11:35:03.939030] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.939210] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49194: ping timer event already removed >[2018-08-18 11:35:03.939248] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.939265] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.939280] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m on vol0-replicate-6 returned error [No such file or directory] >[2018-08-18 11:35:03.939522] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49195: ping timer event already removed >[2018-08-18 11:35:03.939808] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49195: ping timer event already removed >[2018-08-18 11:35:03.940066] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49195: ping timer event already removed >[2018-08-18 11:35:03.940376] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49196: ping timer event already removed >[2018-08-18 11:35:03.940419] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.940645] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49196: ping timer event already removed >[2018-08-18 11:35:03.940688] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.940905] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49196: ping timer event already removed >[2018-08-18 11:35:03.940977] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.941026] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.941058] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m on vol0-replicate-8 returned error [No such file or directory] >[2018-08-18 11:35:03.941218] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49197: ping timer event already removed >[2018-08-18 11:35:03.941287] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.941478] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49197: ping timer event already removed >[2018-08-18 11:35:03.941548] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.941747] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49197: ping timer event already removed >[2018-08-18 11:35:03.941789] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.941807] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.941816] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m on vol0-replicate-9 returned error [No such file or directory] >[2018-08-18 11:35:03.942068] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49198: ping timer event already removed >[2018-08-18 11:35:03.942137] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.942326] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49198: ping timer event already removed >[2018-08-18 11:35:03.942373] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.942587] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49198: ping timer event already removed >[2018-08-18 11:35:03.942662] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.942691] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.942704] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m on vol0-replicate-10 returned error [No such file or directory] >[2018-08-18 11:35:03.942897] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49199: ping timer event already removed >[2018-08-18 11:35:03.942940] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.943152] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49199: ping timer event already removed >[2018-08-18 11:35:03.943198] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.943404] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49199: ping timer event already removed >[2018-08-18 11:35:03.943440] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.943458] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.943467] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m on vol0-replicate-11 returned error [No such file or directory] >[2018-08-18 11:35:03.943711] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49200: ping timer event already removed >[2018-08-18 11:35:03.943955] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49200: ping timer event already removed >[2018-08-18 11:35:03.944025] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.944190] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49200: ping timer event already removed >[2018-08-18 11:35:03.944227] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.944473] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49201: ping timer event already removed >[2018-08-18 11:35:03.944534] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.944673] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa004a920, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.944677] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa004a920, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.944877] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49201: ping timer event already removed >[2018-08-18 11:35:03.944944] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.945004] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-36: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.945027] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbfa004a920, vol0-client-36 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.945039] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-37: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.945048] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-38: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.945062] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbfa004a920, vol0-client-38 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.945058] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbfa004a920, vol0-client-37 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.945218] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-4 >[2018-08-18 11:35:03.945239] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49201: ping timer event already removed >[2018-08-18 11:35:03.945264] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:03.945280] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.945300] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.945311] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m on vol0-replicate-13 returned error [No such file or directory] >[2018-08-18 11:35:03.945542] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49202: ping timer event already removed >[2018-08-18 11:35:03.945655] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.945803] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49202: ping timer event already removed >[2018-08-18 11:35:03.945868] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.946045] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49202: ping timer event already removed >[2018-08-18 11:35:03.946111] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.946150] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.946167] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m on vol0-replicate-14 returned error [No such file or directory] >[2018-08-18 11:35:03.946335] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49203: ping timer event already removed >[2018-08-18 11:35:03.946570] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49203: ping timer event already removed >[2018-08-18 11:35:03.946622] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.946817] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49203: ping timer event already removed >[2018-08-18 11:35:03.947117] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49204: ping timer event already removed >[2018-08-18 11:35:03.947344] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c0131b0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.947363] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49204: ping timer event already removed >[2018-08-18 11:35:03.947635] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49204: ping timer event already removed >[2018-08-18 11:35:03.947692] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-45: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.947717] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c0131b0, vol0-client-45 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.947744] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-46: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.947760] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c0131b0, vol0-client-46 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.947764] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-47: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.947792] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c0131b0, vol0-client-47 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.947967] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49205: ping timer event already removed >[2018-08-18 11:35:03.948262] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49205: ping timer event already removed >[2018-08-18 11:35:03.948500] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49205: ping timer event already removed >[2018-08-18 11:35:03.948826] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49206: ping timer event already removed >[2018-08-18 11:35:03.949083] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49206: ping timer event already removed >[2018-08-18 11:35:03.949166] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.949345] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49206: ping timer event already removed >[2018-08-18 11:35:03.949655] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49207: ping timer event already removed >[2018-08-18 11:35:03.949927] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49207: ping timer event already removed >[2018-08-18 11:35:03.949991] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.950213] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49207: ping timer event already removed >[2018-08-18 11:35:03.950274] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.950569] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49208: ping timer event already removed >[2018-08-18 11:35:03.950614] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.950670] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8000c740, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.950697] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8000c740, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.950892] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49208: ping timer event already removed >[2018-08-18 11:35:03.950976] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-57: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.950998] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-58: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.950999] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf8000c740, vol0-client-57 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.951026] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf8000c740, vol0-client-58 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.951139] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-59: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.951151] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf8000c740, vol0-client-59 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.951196] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49208: ping timer event already removed >[2018-08-18 11:35:03.951261] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.951578] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49209: ping timer event already removed >[2018-08-18 11:35:03.951616] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf7800b000, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.951651] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.951859] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49209: ping timer event already removed >[2018-08-18 11:35:03.951983] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-55: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.951987] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-54: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.952010] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf7800b000, vol0-client-55 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.952012] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-56: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.952020] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf7800b000, vol0-client-54 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.952030] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf7800b000, vol0-client-56 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.952182] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49209: ping timer event already removed >[2018-08-18 11:35:03.952557] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49210: ping timer event already removed >[2018-08-18 11:35:03.952870] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49210: ping timer event already removed >[2018-08-18 11:35:03.952914] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.953137] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49210: ping timer event already removed >[2018-08-18 11:35:03.953509] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49211: ping timer event already removed >[2018-08-18 11:35:03.953786] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49211: ping timer event already removed >[2018-08-18 11:35:03.953851] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.954065] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49211: ping timer event already removed >[2018-08-18 11:35:03.954119] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.954452] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49212: ping timer event already removed >[2018-08-18 11:35:03.954583] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf700047b0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.954613] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf78002dd0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.954653] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c015830, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.954657] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c015830, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.955100] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-69: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.955107] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-70: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.955112] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf8c015830, vol0-client-69 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.955110] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49212: ping timer event already removed >[2018-08-18 11:35:03.955121] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf8c015830, vol0-client-70 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.955143] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-71: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.955153] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf8c015830, vol0-client-71 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.955171] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-63: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.955185] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf78002dd0, vol0-client-63 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.955186] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-65: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.955195] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-64: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.955205] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf78002dd0, vol0-client-65 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.955231] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf78002dd0, vol0-client-64 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.955334] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-60: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.955357] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf700047b0, vol0-client-60 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.955360] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-61: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.955374] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf700047b0, vol0-client-61 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.955385] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-62: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.955397] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf700047b0, vol0-client-62 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.955424] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49212: ping timer event already removed >[2018-08-18 11:35:03.955464] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.955920] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49213: ping timer event already removed >[2018-08-18 11:35:03.955995] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.956209] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa0001140, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.956366] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49213: ping timer event already removed >[2018-08-18 11:35:03.956471] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-66: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.956497] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbfa0001140, vol0-client-66 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.956503] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-67: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.956516] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbfa0001140, vol0-client-67 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.956534] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-68: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.956543] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbfa0001140, vol0-client-68 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.956798] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49213: ping timer event already removed >[2018-08-18 11:35:03.957239] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49214: ping timer event already removed >[2018-08-18 11:35:03.957676] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49214: ping timer event already removed >[2018-08-18 11:35:03.957751] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.958057] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49214: ping timer event already removed >[2018-08-18 11:35:03.958128] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.958596] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf84005850, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.958610] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49215: ping timer event already removed >[2018-08-18 11:35:03.959348] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49215: ping timer event already removed >[2018-08-18 11:35:03.959541] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-78: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.959560] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf84005850, vol0-client-78 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.959566] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-80: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.959580] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf84005850, vol0-client-80 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.959584] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-79: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.959610] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf84005850, vol0-client-79 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.959898] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49215: ping timer event already removed >[2018-08-18 11:35:03.960389] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49216: ping timer event already removed >[2018-08-18 11:35:03.961256] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49216: ping timer event already removed >[2018-08-18 11:35:03.961557] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49216: ping timer event already removed >[2018-08-18 11:35:03.961918] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49217: ping timer event already removed >[2018-08-18 11:35:03.962217] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49217: ping timer event already removed >[2018-08-18 11:35:03.962473] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49217: ping timer event already removed >[2018-08-18 11:35:03.962544] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.962793] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49218: ping timer event already removed >[2018-08-18 11:35:03.963052] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49218: ping timer event already removed >[2018-08-18 11:35:03.963284] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49218: ping timer event already removed >[2018-08-18 11:35:03.963563] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49219: ping timer event already removed >[2018-08-18 11:35:03.963844] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49219: ping timer event already removed >[2018-08-18 11:35:03.964076] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49219: ping timer event already removed >[2018-08-18 11:35:03.964367] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49220: ping timer event already removed >[2018-08-18 11:35:03.964614] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49220: ping timer event already removed >[2018-08-18 11:35:03.964846] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49220: ping timer event already removed >[2018-08-18 11:35:03.965075] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9802fc70, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:03.965212] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49221: ping timer event already removed >[2018-08-18 11:35:03.965431] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-87: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.965445] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9802fc70, vol0-client-87 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.965468] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-89: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.965474] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-88: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.965483] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9802fc70, vol0-client-89 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.965479] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49221: ping timer event already removed >[2018-08-18 11:35:03.965507] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9802fc70, vol0-client-88 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.965849] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49221: ping timer event already removed >[2018-08-18 11:35:03.966291] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49222: ping timer event already removed >[2018-08-18 11:35:03.966546] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49222: ping timer event already removed >[2018-08-18 11:35:03.966834] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49222: ping timer event already removed >[2018-08-18 11:35:03.967265] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49223: ping timer event already removed >[2018-08-18 11:35:03.967588] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49223: ping timer event already removed >[2018-08-18 11:35:03.968140] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49223: ping timer event already removed >[2018-08-18 11:35:03.973901] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-vol0-dht: Found anomalies in /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m (gfid = 905b88d0-eedc-45b8-9d9c-5a6af603d93b). Holes=1 overlaps=0 >[2018-08-18 11:35:03.973925] D [MSGID: 0] [dht-common.c:943:dht_lookup_dir_cbk] 0-vol0-dht: fixing assignment on /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m >[2018-08-18 11:35:03.973972] D [MSGID: 0] [dht-selfheal.c:1879:dht_selfheal_layout_new_directory] 0-vol0-dht: chunk size = 0xffffffff / 274638528 = 15.638619 >[2018-08-18 11:35:03.973986] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-28 >[2018-08-18 11:35:03.973996] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-29 >[2018-08-18 11:35:03.974005] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-3 >[2018-08-18 11:35:03.974013] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-30 >[2018-08-18 11:35:03.974022] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-31 >[2018-08-18 11:35:03.974030] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-32 >[2018-08-18 11:35:03.974038] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-33 >[2018-08-18 11:35:03.974047] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-34 >[2018-08-18 11:35:03.974055] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-35 >[2018-08-18 11:35:03.974063] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-4 >[2018-08-18 11:35:03.974071] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-5 >[2018-08-18 11:35:03.974079] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-6 >[2018-08-18 11:35:03.974087] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-7 >[2018-08-18 11:35:03.974095] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-8 >[2018-08-18 11:35:03.974103] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-9 >[2018-08-18 11:35:03.974111] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-0 >[2018-08-18 11:35:03.974119] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-1 >[2018-08-18 11:35:03.974127] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-10 >[2018-08-18 11:35:03.974140] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-11 >[2018-08-18 11:35:03.974148] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-12 >[2018-08-18 11:35:03.974156] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-13 >[2018-08-18 11:35:03.974164] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-14 >[2018-08-18 11:35:03.974172] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-15 >[2018-08-18 11:35:03.974180] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-16 >[2018-08-18 11:35:03.974188] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-17 >[2018-08-18 11:35:03.974196] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-18 >[2018-08-18 11:35:03.974204] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-19 >[2018-08-18 11:35:03.974212] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-2 >[2018-08-18 11:35:03.974220] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-20 >[2018-08-18 11:35:03.974228] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-21 >[2018-08-18 11:35:03.974236] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-22 >[2018-08-18 11:35:03.974244] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-23 >[2018-08-18 11:35:03.974252] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-24 >[2018-08-18 11:35:03.974260] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-25 >[2018-08-18 11:35:03.974268] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-26 >[2018-08-18 11:35:03.974276] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-27 >[2018-08-18 11:35:03.974505] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-0: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.974515] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-1: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.974523] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbf9403a560, vol0-client-0 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.974535] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbf9403a560, vol0-client-1 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:03.974709] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-2: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:03.974744] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbf9403a560, vol0-client-2 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.039549] D [MSGID: 0] [dht-selfheal.c:1206:dht_selfheal_dir_mkdir_setacl] 0-vol0-dht: ACL_DEFAULT xattr not present >[2018-08-18 11:35:04.039569] D [MSGID: 0] [dht-selfheal.c:1219:dht_selfheal_dir_mkdir_setacl] 0-vol0-dht: ACL_ACCESS xattr not present >[2018-08-18 11:35:04.039577] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m on subvol vol0-replicate-1 >[2018-08-18 11:35:04.039651] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m on subvol vol0-replicate-10 >[2018-08-18 11:35:04.039709] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m on subvol vol0-replicate-11 >[2018-08-18 11:35:04.039760] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-1: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.039760] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m on subvol vol0-replicate-13 >[2018-08-18 11:35:04.039818] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-10: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.039850] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m on subvol vol0-replicate-14 >[2018-08-18 11:35:04.039875] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-11: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.039888] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-4: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.039878] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-3: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.039918] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc067640, vol0-client-4 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.039915] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m on subvol vol0-replicate-2 >[2018-08-18 11:35:04.039939] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc067640, vol0-client-3 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.039946] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-30: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.039942] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-5: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.039948] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-13: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.039956] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc085190, vol0-client-30 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.039989] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc067640, vol0-client-5 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.039984] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-31: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.039984] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m on subvol vol0-replicate-3 >[2018-08-18 11:35:04.040012] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 11:35:04.040015] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc085190, vol0-client-31 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040021] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-33: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040036] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc284b70, vol0-client-33 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040026] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-32: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040049] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-14: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.040069] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m on subvol vol0-replicate-6 >[2018-08-18 11:35:04.040077] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-2: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.040085] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-39: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040086] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc085190, vol0-client-32 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040103] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc000d90, vol0-client-39 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040061] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-34: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040127] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-40: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040130] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m on subvol vol0-replicate-8 >[2018-08-18 11:35:04.040131] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-35: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040132] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:04.040145] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-41: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040132] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc284b70, vol0-client-34 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040170] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc000d90, vol0-client-41 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040139] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc000d90, vol0-client-40 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040189] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc284b70, vol0-client-35 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040191] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m on subvol vol0-replicate-9 >[2018-08-18 11:35:04.040191] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 11:35:04.040208] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:04.040223] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-3: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.040236] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-replicate-1 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040275] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-43: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040289] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc002d80, vol0-client-43 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040281] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m, gfid = 905b88d0-eedc-45b8-9d9c-5a6af603d93b [File exists] >[2018-08-18 11:35:04.040279] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-42: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040292] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-6: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040293] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-44: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040318] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc061b20, vol0-client-6 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040311] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-7: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040310] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-8: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040340] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc061b20, vol0-client-8 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040332] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc002d80, vol0-client-42 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040343] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc061b20, vol0-client-7 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040345] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc002d80, vol0-client-44 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040399] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 11:35:04.040401] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-6: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.040401] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-8: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.040402] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 11:35:04.040421] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-replicate-13 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040431] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-9: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040456] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc02b770, vol0-client-9 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040458] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m, gfid = 905b88d0-eedc-45b8-9d9c-5a6af603d93b [File exists] >[2018-08-18 11:35:04.040474] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-replicate-10 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040481] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-9: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.040486] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-10: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040490] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m, gfid = 905b88d0-eedc-45b8-9d9c-5a6af603d93b [File exists] >[2018-08-18 11:35:04.040505] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-replicate-11 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040514] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc02b770, vol0-client-10 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040513] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-11: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040525] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m, gfid = 905b88d0-eedc-45b8-9d9c-5a6af603d93b [File exists] >[2018-08-18 11:35:04.040525] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-24: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040546] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc02b770, vol0-client-11 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040542] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-25: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040545] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-18: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040580] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 11:35:04.040576] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc05fbb0, vol0-client-25 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040591] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-20: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040589] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc07cb10, vol0-client-18 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040641] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc07cb10, vol0-client-20 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040581] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-19: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040652] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-26: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040606] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc05fbb0, vol0-client-24 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040676] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc07cb10, vol0-client-19 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040680] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-replicate-14 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040691] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 11:35:04.040695] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m, gfid = 905b88d0-eedc-45b8-9d9c-5a6af603d93b [File exists] >[2018-08-18 11:35:04.040683] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc05fbb0, vol0-client-26 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040696] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-replicate-2 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040715] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 11:35:04.040719] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m, gfid = 905b88d0-eedc-45b8-9d9c-5a6af603d93b [File exists] >[2018-08-18 11:35:04.040720] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-28: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040722] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-27: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040737] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-29: remote operation failed. Path: /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m [File exists] >[2018-08-18 11:35:04.040763] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc6877d0, vol0-client-28 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040761] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc6877d0, vol0-client-29 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040768] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfbc6877d0, vol0-client-27 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040794] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 11:35:04.040810] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-replicate-3 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040833] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m, gfid = 905b88d0-eedc-45b8-9d9c-5a6af603d93b [File exists] >[2018-08-18 11:35:04.040841] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-replicate-6 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040842] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-replicate-8 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040864] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m, gfid = 905b88d0-eedc-45b8-9d9c-5a6af603d93b [File exists] >[2018-08-18 11:35:04.040955] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 11:35:04.040865] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m, gfid = 905b88d0-eedc-45b8-9d9c-5a6af603d93b [File exists] >[2018-08-18 11:35:04.040954] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa84901a0, vol0-replicate-9 returned -1 error: File exists [File exists] >[2018-08-18 11:35:04.040992] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m, gfid = 905b88d0-eedc-45b8-9d9c-5a6af603d93b [File exists] >[2018-08-18 11:35:04.042523] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-0: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.042688] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-12: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.042747] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 11:35:04.042797] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-1: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.042850] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-15: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.042896] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.042970] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-17: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.042976] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.042988] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 11:35:04.043016] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.043054] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-18: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.043101] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-19: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.043111] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 11:35:04.043113] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.043175] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 11:35:04.043206] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.043213] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 11:35:04.043204] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-20: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.043286] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.043365] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 11:35:04.043376] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-22: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.043381] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:04.043436] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:04.043460] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-23: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.043487] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 11:35:04.043494] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 11:35:04.043518] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.043556] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 11:35:04.043557] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-25: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.043630] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 11:35:04.043638] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-26: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.043669] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-20: lk op is for a transaction >[2018-08-18 11:35:04.043692] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-27: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.043721] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 11:35:04.043749] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-2: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.043749] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-28: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.043820] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-22: lk op is for a transaction >[2018-08-18 11:35:04.043820] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-29: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.043894] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-30: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.043914] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-23: lk op is for a transaction >[2018-08-18 11:35:04.043972] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.043985] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:04.044007] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 11:35:04.044022] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-32: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.044064] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-26: lk op is for a transaction >[2018-08-18 11:35:04.044084] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-33: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.044125] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-34: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.044152] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-27: lk op is for a transaction >[2018-08-18 11:35:04.044160] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-35: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.044185] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 11:35:04.044188] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 11:35:04.044212] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.044231] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 11:35:04.044264] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-5: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.044290] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-30: lk op is for a transaction >[2018-08-18 11:35:04.044298] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-3: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.044342] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.044363] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 11:35:04.044428] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 11:35:04.044418] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 11:35:04.044487] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 11:35:04.044515] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 11:35:04.044558] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:04.044577] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-6: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.044603] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 11:35:04.044593] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 11:35:04.044646] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 11:35:04.044654] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-8: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.044710] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-9: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.044768] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 11:35:04.044854] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 11:35:04.045038] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 11:35:04.045106] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-0: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.045126] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfcc02a8d0, vol0-client-0 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.045134] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-2: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.045135] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-1: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.045153] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfcc02a8d0, vol0-client-2 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.045164] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfcc02a8d0, vol0-client-1 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.056157] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7fbfa0015d20) as it exceeds the maximum available buffer size >[2018-08-18 11:35:04.057494] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.057531] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=95372770-e4d0-42bb-8548-d239e992ad24, gen=0): picking the request for winding >[2018-08-18 11:35:04.057557] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=95372770-e4d0-42bb-8548-d239e992ad24, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:04.057827] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.075982] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=95372770-e4d0-42bb-8548-d239e992ad24, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:04.076103] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=95372770-e4d0-42bb-8548-d239e992ad24, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.076206] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.096161] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-10 >[2018-08-18 11:35:04.096289] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88 on vol0-replicate-25 >[2018-08-18 11:35:04.096446] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096452] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096473] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096498] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096509] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 with op_ret -1 [No such file or directory] >[2018-08-18 11:35:04.096520] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 missing on subvol vol0-replicate-10 >[2018-08-18 11:35:04.096538] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:04.096564] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096601] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096612] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096633] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096643] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88 with op_ret -1 [No such file or directory] >[2018-08-18 11:35:04.096652] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88 missing on subvol vol0-replicate-25 >[2018-08-18 11:35:04.096659] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:04.096804] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096806] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096858] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096852] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096876] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096892] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096896] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096898] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096927] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096929] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096950] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096934] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096954] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096966] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096904] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096975] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-0 >[2018-08-18 11:35:04.096977] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-1 >[2018-08-18 11:35:04.096994] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097010] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.096997] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-0 >[2018-08-18 11:35:04.097002] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097016] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097024] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097043] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-1 >[2018-08-18 11:35:04.097043] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097040] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097053] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097047] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097073] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097081] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097088] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097086] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097089] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097079] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097102] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-3 >[2018-08-18 11:35:04.097112] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097127] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097082] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-2 >[2018-08-18 11:35:04.097127] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097140] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097140] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097132] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097167] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-4 >[2018-08-18 11:35:04.097166] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097170] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097166] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097175] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097166] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-2 >[2018-08-18 11:35:04.097179] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-3 >[2018-08-18 11:35:04.097193] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097202] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097215] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-5 >[2018-08-18 11:35:04.097224] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097229] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097229] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097217] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097235] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097253] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097261] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097266] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097262] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097276] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097277] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-6 >[2018-08-18 11:35:04.097277] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097290] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-4 >[2018-08-18 11:35:04.097300] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097309] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-5 >[2018-08-18 11:35:04.097320] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097320] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097329] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097345] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097359] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-7 >[2018-08-18 11:35:04.097344] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097368] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097383] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097390] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097389] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097396] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097392] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097392] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097411] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097436] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-8 >[2018-08-18 11:35:04.097436] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097436] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097447] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-7 >[2018-08-18 11:35:04.097440] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-6 >[2018-08-18 11:35:04.097456] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097469] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097469] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097472] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097493] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097504] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097497] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097510] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-9 >[2018-08-18 11:35:04.097518] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097512] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097526] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-8 >[2018-08-18 11:35:04.097566] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097567] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097584] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097604] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097584] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097588] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097639] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097644] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097653] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097640] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097664] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-11 >[2018-08-18 11:35:04.097679] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097676] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-9 >[2018-08-18 11:35:04.097685] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097708] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097720] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097729] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-12 >[2018-08-18 11:35:04.097758] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097780] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097786] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097752] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097788] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097806] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097824] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097818] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097813] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097824] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097814] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097827] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097812] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097849] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-11 >[2018-08-18 11:35:04.097814] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097856] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097865] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097878] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097879] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097872] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-10 >[2018-08-18 11:35:04.097890] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097898] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097890] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097896] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-12 >[2018-08-18 11:35:04.097907] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-13 >[2018-08-18 11:35:04.097911] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097914] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-14 >[2018-08-18 11:35:04.097934] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097944] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097931] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097952] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-13 >[2018-08-18 11:35:04.097936] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097962] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097936] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097961] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097988] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.097995] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098000] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098009] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098023] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098014] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098033] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-14 >[2018-08-18 11:35:04.098013] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098036] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098017] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098040] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-16 >[2018-08-18 11:35:04.098057] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098069] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098057] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-15 >[2018-08-18 11:35:04.098081] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098079] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-17 >[2018-08-18 11:35:04.098088] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098096] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098090] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098109] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-15 >[2018-08-18 11:35:04.098125] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098137] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098147] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098143] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098151] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098162] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-18 >[2018-08-18 11:35:04.098170] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098178] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-16 >[2018-08-18 11:35:04.098199] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098215] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098221] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098221] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098226] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098246] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098239] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098251] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098252] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098270] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-19 >[2018-08-18 11:35:04.098272] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098275] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098285] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098278] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098305] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098304] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-17 >[2018-08-18 11:35:04.098312] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-20 >[2018-08-18 11:35:04.098304] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098323] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098332] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098337] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098340] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098343] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098358] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-18 >[2018-08-18 11:35:04.098360] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098391] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098400] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098415] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098418] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-19 >[2018-08-18 11:35:04.098424] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098429] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098426] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098445] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098460] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098475] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098472] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098482] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-22 >[2018-08-18 11:35:04.098478] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098478] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098492] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098501] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098480] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098502] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-20 >[2018-08-18 11:35:04.098508] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098515] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-21 >[2018-08-18 11:35:04.098539] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098544] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098549] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-23 >[2018-08-18 11:35:04.098567] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098569] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098583] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098587] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098603] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098592] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098614] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-24 >[2018-08-18 11:35:04.098610] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098593] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098622] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098623] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098639] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098643] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098641] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098627] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098661] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098669] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-21 >[2018-08-18 11:35:04.098671] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-22 >[2018-08-18 11:35:04.098688] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098690] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098702] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098715] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098713] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098718] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-23 >[2018-08-18 11:35:04.098731] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098737] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098735] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098747] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098755] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-25 >[2018-08-18 11:35:04.098756] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098775] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098787] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098789] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098812] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098802] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-24 >[2018-08-18 11:35:04.098819] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098819] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-27 >[2018-08-18 11:35:04.098816] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098839] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098799] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098840] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098835] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098842] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-26 >[2018-08-18 11:35:04.098833] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098868] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098875] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098889] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098889] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-26 >[2018-08-18 11:35:04.098897] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-28 >[2018-08-18 11:35:04.098924] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098914] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098948] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098961] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098964] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098975] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098964] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.098981] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-29 >[2018-08-18 11:35:04.099007] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099005] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099023] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099012] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099024] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099046] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099068] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099072] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099076] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099051] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099091] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-30 >[2018-08-18 11:35:04.099097] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-27 >[2018-08-18 11:35:04.099099] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099104] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099102] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099096] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099124] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099089] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099129] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099120] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099136] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099122] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-29 >[2018-08-18 11:35:04.099145] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-31 >[2018-08-18 11:35:04.099147] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099165] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-28 >[2018-08-18 11:35:04.099153] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099180] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099179] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099183] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099190] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099192] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-30 >[2018-08-18 11:35:04.099191] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099221] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099225] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099227] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099242] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099233] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099253] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-31 >[2018-08-18 11:35:04.099255] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099259] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099271] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099280] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-32 >[2018-08-18 11:35:04.099274] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099287] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099295] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099275] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099309] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099297] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099310] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099318] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-32 >[2018-08-18 11:35:04.099319] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-10 >[2018-08-18 11:35:04.099326] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099346] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099361] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099359] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099375] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099382] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-34 >[2018-08-18 11:35:04.099374] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099387] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099391] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099391] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099385] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-33 >[2018-08-18 11:35:04.099414] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099416] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099421] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099415] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099423] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099435] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-33 >[2018-08-18 11:35:04.099443] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099441] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-25 >[2018-08-18 11:35:04.099451] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-35 >[2018-08-18 11:35:04.099441] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099459] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-10 cached_subvol null >[2018-08-18 11:35:04.099460] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-34 >[2018-08-18 11:35:04.099466] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 >[2018-08-18 11:35:04.099474] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099481] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099490] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099498] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099499] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099506] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8014070, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099517] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099571] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099583] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099591] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88) from subvol vol0-replicate-35 >[2018-08-18 11:35:04.099604] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-25 cached_subvol null >[2018-08-18 11:35:04.099614] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88 >[2018-08-18 11:35:04.099621] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099629] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099637] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099644] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099651] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c04c940, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.099807] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-0': avail_percent is: 96.00 and avail_space is: 7682559471616 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.099858] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-1': avail_percent is: 95.00 and avail_space is: 7649225949184 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.099890] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-2': avail_percent is: 96.00 and avail_space is: 7679628845056 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.099931] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-3': avail_percent is: 95.00 and avail_space is: 7676722438144 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.099950] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-0': avail_percent is: 96.00 and avail_space is: 7682559471616 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.099971] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-4': avail_percent is: 96.00 and avail_space is: 7679877980160 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100022] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-1': avail_percent is: 95.00 and avail_space is: 7649225949184 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100036] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-5': avail_percent is: 95.00 and avail_space is: 7672041328640 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100077] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-6': avail_percent is: 94.00 and avail_space is: 7567699996672 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100090] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-2': avail_percent is: 96.00 and avail_space is: 7679628845056 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100103] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-7': avail_percent is: 95.00 and avail_space is: 7599561781248 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100140] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-3': avail_percent is: 95.00 and avail_space is: 7676722438144 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100143] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-8': avail_percent is: 95.00 and avail_space is: 7669932085248 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100180] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-9': avail_percent is: 95.00 and avail_space is: 7675197493248 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100211] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-4': avail_percent is: 96.00 and avail_space is: 7679877980160 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100222] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-10': avail_percent is: 94.00 and avail_space is: 7587263074304 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100254] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-11': avail_percent is: 95.00 and avail_space is: 7666176073728 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100279] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-5': avail_percent is: 95.00 and avail_space is: 7672041328640 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100307] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-12': avail_percent is: 95.00 and avail_space is: 7662336405504 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100325] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-6': avail_percent is: 94.00 and avail_space is: 7567699996672 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100385] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-7': avail_percent is: 95.00 and avail_space is: 7599561781248 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100393] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-13': avail_percent is: 95.00 and avail_space is: 7671819669504 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100417] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-14': avail_percent is: 95.00 and avail_space is: 7665779671040 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100451] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-8': avail_percent is: 95.00 and avail_space is: 7669932085248 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100460] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-15': avail_percent is: 95.00 and avail_space is: 7654391312384 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100500] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-9': avail_percent is: 95.00 and avail_space is: 7675197493248 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100503] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-17': avail_percent is: 95.00 and avail_space is: 7669472276480 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100536] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-16': avail_percent is: 95.00 and avail_space is: 7678806634496 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100571] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-10': avail_percent is: 94.00 and avail_space is: 7587263074304 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100595] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-18': avail_percent is: 95.00 and avail_space is: 7615159742464 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100635] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-19': avail_percent is: 95.00 and avail_space is: 7660364656640 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100660] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-11': avail_percent is: 95.00 and avail_space is: 7666176073728 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100666] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-20': avail_percent is: 95.00 and avail_space is: 7610263240704 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100687] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-12': avail_percent is: 95.00 and avail_space is: 7662336405504 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100726] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-21': avail_percent is: 95.00 and avail_space is: 7663388819456 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100750] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-22': avail_percent is: 95.00 and avail_space is: 7668074450944 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100776] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-13': avail_percent is: 95.00 and avail_space is: 7671819669504 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100824] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-23': avail_percent is: 95.00 and avail_space is: 7669669240832 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100845] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-14': avail_percent is: 95.00 and avail_space is: 7665779671040 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100858] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-24': avail_percent is: 95.00 and avail_space is: 7660816109568 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100880] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-15': avail_percent is: 95.00 and avail_space is: 7654391312384 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100913] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-25': avail_percent is: 95.00 and avail_space is: 7644139499520 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100929] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-26': avail_percent is: 95.00 and avail_space is: 7655957176320 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.100969] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-16': avail_percent is: 95.00 and avail_space is: 7678806634496 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101008] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-27': avail_percent is: 95.00 and avail_space is: 7652166385664 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101024] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-17': avail_percent is: 95.00 and avail_space is: 7669472276480 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101033] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-28': avail_percent is: 95.00 and avail_space is: 7652181016576 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101067] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-29': avail_percent is: 95.00 and avail_space is: 7639365349376 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101093] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-18': avail_percent is: 95.00 and avail_space is: 7615159742464 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101134] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-30': avail_percent is: 95.00 and avail_space is: 7675637047296 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101158] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-31': avail_percent is: 95.00 and avail_space is: 7678164856832 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101179] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-19': avail_percent is: 95.00 and avail_space is: 7660364656640 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101185] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-32': avail_percent is: 95.00 and avail_space is: 7659078995968 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101233] D [MSGID: 0] [dht-common.c:6923:dht_create_wind_to_avail_subvol] 0-vol0-dht: creating /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 on vol0-replicate-10 >[2018-08-18 11:35:04.101236] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-20': avail_percent is: 95.00 and avail_space is: 7610263240704 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101274] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-33': avail_percent is: 95.00 and avail_space is: 7676538961920 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101309] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-21': avail_percent is: 95.00 and avail_space is: 7663388819456 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101328] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-35': avail_percent is: 95.00 and avail_space is: 7661847982080 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101344] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-34': avail_percent is: 95.00 and avail_space is: 7665343188992 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101382] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-22': avail_percent is: 95.00 and avail_space is: 7668074450944 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101424] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-23': avail_percent is: 95.00 and avail_space is: 7669669240832 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101424] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-10: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.101477] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-24': avail_percent is: 95.00 and avail_space is: 7660816109568 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101529] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-25': avail_percent is: 95.00 and avail_space is: 7644139499520 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101585] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-26': avail_percent is: 95.00 and avail_space is: 7655957176320 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101655] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-27': avail_percent is: 95.00 and avail_space is: 7652166385664 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101706] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-28': avail_percent is: 95.00 and avail_space is: 7652181016576 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101746] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-29': avail_percent is: 95.00 and avail_space is: 7639365349376 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101857] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-30': avail_percent is: 95.00 and avail_space is: 7675637047296 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101895] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-31': avail_percent is: 95.00 and avail_space is: 7678164856832 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.101941] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-32': avail_percent is: 95.00 and avail_space is: 7659078995968 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.102010] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-33': avail_percent is: 95.00 and avail_space is: 7676538961920 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.102029] D [MSGID: 0] [dht-common.c:6923:dht_create_wind_to_avail_subvol] 0-vol0-dht: creating /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88 on vol0-replicate-25 >[2018-08-18 11:35:04.102067] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-34': avail_percent is: 95.00 and avail_space is: 7665343188992 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.102121] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-35': avail_percent is: 95.00 and avail_space is: 7661847982080 and avail_inodes is: 99.00 >[2018-08-18 11:35:04.102245] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-25: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.102531] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-25 >[2018-08-18 11:35:04.102573] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 11:35:04.127674] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-10 >[2018-08-18 11:35:04.127708] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:04.236787] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7fbfb00c24b0) as it exceeds the maximum available buffer size >[2018-08-18 11:35:04.238254] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.238292] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=165daf77-45ae-451a-b5a4-415ab86aedeb, gen=0): picking the request for winding >[2018-08-18 11:35:04.238313] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=165daf77-45ae-451a-b5a4-415ab86aedeb, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:04.238686] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-25: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.256644] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=165daf77-45ae-451a-b5a4-415ab86aedeb, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:04.256769] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=165daf77-45ae-451a-b5a4-415ab86aedeb, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.256878] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.288625] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7fbfa000d030) as it exceeds the maximum available buffer size >[2018-08-18 11:35:04.292313] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7fbfa8064430) as it exceeds the maximum available buffer size >[2018-08-18 11:35:04.292427] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.292457] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=e40b648a-8f88-4808-9583-9143d5a4444f, gen=0): picking the request for winding >[2018-08-18 11:35:04.292472] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=e40b648a-8f88-4808-9583-9143d5a4444f, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:04.292776] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.293490] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.293517] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=95372770-e4d0-42bb-8548-d239e992ad24, gen=0): picking the request for winding >[2018-08-18 11:35:04.293535] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=95372770-e4d0-42bb-8548-d239e992ad24, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:04.293589] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.293617] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:04.293646] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7fbfa0015d20) allocated with standard calloc() >[2018-08-18 11:35:04.316786] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=95372770-e4d0-42bb-8548-d239e992ad24, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:04.316941] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=95372770-e4d0-42bb-8548-d239e992ad24, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.317080] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.319672] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=e40b648a-8f88-4808-9583-9143d5a4444f, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:04.319785] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=e40b648a-8f88-4808-9583-9143d5a4444f, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.319883] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.335712] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.335740] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=95372770-e4d0-42bb-8548-d239e992ad24, gen=0): picking the request for winding >[2018-08-18 11:35:04.335992] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=95372770-e4d0-42bb-8548-d239e992ad24, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.336072] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:04.336223] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7fbfa8064430) allocated with standard calloc() >[2018-08-18 11:35:04.346483] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001 on vol0-replicate-35 >[2018-08-18 11:35:04.346794] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.346828] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.346835] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.346870] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.346882] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001 with op_ret -1 [No such file or directory] >[2018-08-18 11:35:04.346893] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001 missing on subvol vol0-replicate-35 >[2018-08-18 11:35:04.346902] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:04.347171] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347201] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347210] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347249] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347251] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347276] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-0 >[2018-08-18 11:35:04.347279] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347305] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347315] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347356] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347358] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347377] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-1 >[2018-08-18 11:35:04.347389] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347401] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347416] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-2 >[2018-08-18 11:35:04.347435] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347434] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347472] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347505] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347520] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347520] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-3 >[2018-08-18 11:35:04.347521] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347548] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347561] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347568] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-4 >[2018-08-18 11:35:04.347618] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347625] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347646] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347658] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347666] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-5 >[2018-08-18 11:35:04.347703] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347715] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347723] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347747] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347755] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-6 >[2018-08-18 11:35:04.347808] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347807] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347824] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347839] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347847] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-7 >[2018-08-18 11:35:04.347879] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347883] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347937] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347950] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347968] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347969] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347976] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.347994] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-8 >[2018-08-18 11:35:04.348001] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348011] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-9 >[2018-08-18 11:35:04.348037] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348037] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348098] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348103] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348113] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348112] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348138] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-10 >[2018-08-18 11:35:04.348174] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348198] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348201] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348216] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348217] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-11 >[2018-08-18 11:35:04.348244] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348248] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348257] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348264] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348274] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-12 >[2018-08-18 11:35:04.348291] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348310] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348318] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-13 >[2018-08-18 11:35:04.348361] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348362] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348380] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348394] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348401] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-14 >[2018-08-18 11:35:04.348422] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348445] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348458] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348476] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348484] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-15 >[2018-08-18 11:35:04.348504] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348510] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348538] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348551] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348559] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-16 >[2018-08-18 11:35:04.348572] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348611] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348619] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348650] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348662] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-17 >[2018-08-18 11:35:04.348665] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348682] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348713] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348708] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348734] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348739] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348757] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-18 >[2018-08-18 11:35:04.348768] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348803] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348820] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-19 >[2018-08-18 11:35:04.348835] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348840] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348848] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348863] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348872] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-20 >[2018-08-18 11:35:04.348906] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348917] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348970] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348972] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348991] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.348993] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349006] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-21 >[2018-08-18 11:35:04.349023] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349041] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349044] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349053] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-22 >[2018-08-18 11:35:04.349074] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349151] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349156] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349188] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349194] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349202] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349225] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-23 >[2018-08-18 11:35:04.349237] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349260] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349262] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-24 >[2018-08-18 11:35:04.349289] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349291] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349319] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349329] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349335] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-25 >[2018-08-18 11:35:04.349373] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349374] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349392] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349400] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-26 >[2018-08-18 11:35:04.349431] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349460] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349471] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349483] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349490] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-27 >[2018-08-18 11:35:04.349553] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349553] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349582] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349595] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349615] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-28 >[2018-08-18 11:35:04.349628] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349643] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349658] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349671] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349680] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-29 >[2018-08-18 11:35:04.349738] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349742] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349742] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349763] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349776] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349778] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349791] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-30 >[2018-08-18 11:35:04.349816] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349829] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349861] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-31 >[2018-08-18 11:35:04.349863] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349871] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349888] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349900] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349907] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-32 >[2018-08-18 11:35:04.349926] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349963] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349978] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.349996] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.350010] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.350006] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.350007] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-33 >[2018-08-18 11:35:04.350052] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.350066] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.350063] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.350063] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.350089] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-34 >[2018-08-18 11:35:04.350103] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.350119] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.350127] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-35 >[2018-08-18 11:35:04.350135] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-35 cached_subvol null >[2018-08-18 11:35:04.350141] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001 >[2018-08-18 11:35:04.350147] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x2029970, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.350155] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.350164] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.350171] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.350180] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.350843] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m returned with op_ret 0 >[2018-08-18 11:35:04.354155] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.354180] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m returned with op_ret 0" repeated 35 times between [2018-08-18 11:35:04.350843] and [2018-08-18 11:35:04.353775] >[2018-08-18 11:35:04.354180] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=95372770-e4d0-42bb-8548-d239e992ad24, gen=0): picking the request for winding >[2018-08-18 11:35:04.354222] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88 at vol0-replicate-4 >[2018-08-18 11:35:04.354438] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=95372770-e4d0-42bb-8548-d239e992ad24, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.354584] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88 returned with op_ret 0 >[2018-08-18 11:35:04.354725] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001 on vol0-replicate-35 >[2018-08-18 11:35:04.354964] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.354970] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.354994] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355015] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355024] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001 with op_ret -1 [No such file or directory] >[2018-08-18 11:35:04.355032] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001 missing on subvol vol0-replicate-35 >[2018-08-18 11:35:04.355039] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:04.355273] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355282] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355307] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355325] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355338] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-0 >[2018-08-18 11:35:04.355346] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355374] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355384] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355418] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355415] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355441] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-1 >[2018-08-18 11:35:04.355452] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355507] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355514] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355518] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355538] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355552] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355554] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355559] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-2 >[2018-08-18 11:35:04.355560] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355586] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-3 >[2018-08-18 11:35:04.355608] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355622] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355634] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355646] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355652] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355677] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355678] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355684] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355682] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355704] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-5 >[2018-08-18 11:35:04.355712] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-4 >[2018-08-18 11:35:04.355715] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355718] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355729] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355731] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355749] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-6 >[2018-08-18 11:35:04.355753] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355773] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355779] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355781] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-7 >[2018-08-18 11:35:04.355794] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355817] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355844] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355855] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-8 >[2018-08-18 11:35:04.355862] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355879] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355886] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355899] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355907] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-9 >[2018-08-18 11:35:04.355929] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355946] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355963] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355975] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.355982] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-10 >[2018-08-18 11:35:04.356008] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356023] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356033] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356057] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356060] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356073] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-11 >[2018-08-18 11:35:04.356132] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356148] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356150] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356149] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356171] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356201] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356203] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-13 >[2018-08-18 11:35:04.356203] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356238] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356250] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356251] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356257] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356274] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356277] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356289] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-14 >[2018-08-18 11:35:04.356293] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356294] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-15 >[2018-08-18 11:35:04.356308] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356328] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356340] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356347] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-16 >[2018-08-18 11:35:04.356356] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356376] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356401] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356416] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356426] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-17 >[2018-08-18 11:35:04.356450] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356445] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356478] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356505] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356511] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356523] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356530] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-18 >[2018-08-18 11:35:04.356561] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356566] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356566] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356577] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356615] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356616] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356635] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356637] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-19 >[2018-08-18 11:35:04.356643] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-20 >[2018-08-18 11:35:04.356656] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356658] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356662] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356687] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356688] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356683] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356698] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356706] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356706] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-21 >[2018-08-18 11:35:04.356717] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-12 >[2018-08-18 11:35:04.356719] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356743] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-22 >[2018-08-18 11:35:04.356754] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356743] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356758] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356778] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356791] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-23 >[2018-08-18 11:35:04.356820] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356827] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356824] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356843] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356850] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-24 >[2018-08-18 11:35:04.356865] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356876] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356901] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356915] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356923] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-25 >[2018-08-18 11:35:04.356926] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356948] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356964] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356981] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.356999] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-26 >[2018-08-18 11:35:04.357004] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357011] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357034] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357047] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357057] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-27 >[2018-08-18 11:35:04.357062] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357076] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357097] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357113] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357120] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-28 >[2018-08-18 11:35:04.357122] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357147] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357149] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357163] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357171] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-29 >[2018-08-18 11:35:04.357196] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357204] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357221] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357238] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357246] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-30 >[2018-08-18 11:35:04.357266] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357266] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357292] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357301] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357306] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357310] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-31 >[2018-08-18 11:35:04.357325] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357339] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357351] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357358] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-32 >[2018-08-18 11:35:04.357380] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357384] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357382] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357418] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357432] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357434] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-33 >[2018-08-18 11:35:04.357433] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357435] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357458] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357465] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-34 >[2018-08-18 11:35:04.357490] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357498] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357513] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357528] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357540] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001) from subvol vol0-replicate-35 >[2018-08-18 11:35:04.357552] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-35 cached_subvol null >[2018-08-18 11:35:04.357562] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001 >[2018-08-18 11:35:04.357572] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357579] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357588] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357602] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357619] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa8067f50, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.357730] I [MSGID: 109066] [dht-rename.c:1610:dht_rename] 0-vol0-dht: renaming /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001_bfafdf58a2da11e8a3196c92bf5c1b88 (hash=vol0-replicate-4/cache=vol0-replicate-4) => /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001 (hash=vol0-replicate-35/cache=<nul>) >[2018-08-18 11:35:04.358395] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-35: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.381841] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7fbfa8434490) as it exceeds the maximum available buffer size >[2018-08-18 11:35:04.382855] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.382880] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=165daf77-45ae-451a-b5a4-415ab86aedeb, gen=0): picking the request for winding >[2018-08-18 11:35:04.382894] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=165daf77-45ae-451a-b5a4-415ab86aedeb, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:04.382955] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-25: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.382969] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 11:35:04.382995] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7fbfb00c24b0) allocated with standard calloc() >[2018-08-18 11:35:04.387790] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 11:35:04.388129] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-4: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.388885] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:04.389258] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-35: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.389919] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 11:35:04.390719] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-4: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.391212] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:04.391439] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-35: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.391801] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 11:35:04.392321] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.392344] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=95372770-e4d0-42bb-8548-d239e992ad24, gen=0): picking the request for winding >[2018-08-18 11:35:04.392368] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001 at vol0-replicate-4 >[2018-08-18 11:35:04.392559] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=95372770-e4d0-42bb-8548-d239e992ad24, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.392808] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001 returned with op_ret 0 >[2018-08-18 11:35:04.394121] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.394510] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:04.394714] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.394766] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=1, gfid=165daf77-45ae-451a-b5a4-415ab86aedeb. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=165daf77-45ae-451a-b5a4-415ab86aedeb, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:04.394815] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.394832] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=95372770-e4d0-42bb-8548-d239e992ad24, gen=0): picking the request for winding >[2018-08-18 11:35:04.394853] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001 at vol0-replicate-4 >[2018-08-18 11:35:04.395043] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=95372770-e4d0-42bb-8548-d239e992ad24, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.395295] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001 returned with op_ret 0 >[2018-08-18 11:35:04.395663] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.396003] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:04.396171] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.396187] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=95372770-e4d0-42bb-8548-d239e992ad24, gen=0): picking the request for winding >[2018-08-18 11:35:04.396207] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001 at vol0-replicate-4 >[2018-08-18 11:35:04.396404] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=95372770-e4d0-42bb-8548-d239e992ad24, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.396738] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001 returned with op_ret 0 >[2018-08-18 11:35:04.397116] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.397375] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:04.397545] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.397561] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=95372770-e4d0-42bb-8548-d239e992ad24, gen=0): picking the request for winding >[2018-08-18 11:35:04.397582] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001 at vol0-replicate-4 >[2018-08-18 11:35:04.397775] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=95372770-e4d0-42bb-8548-d239e992ad24, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.397909] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0001 returned with op_ret 0 >[2018-08-18 11:35:04.398138] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.398362] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:04.399496] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=165daf77-45ae-451a-b5a4-415ab86aedeb, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:04.399617] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=165daf77-45ae-451a-b5a4-415ab86aedeb, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.399723] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.399739] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=165daf77-45ae-451a-b5a4-415ab86aedeb, gen=1): picking the request for winding >[2018-08-18 11:35:04.399951] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=165daf77-45ae-451a-b5a4-415ab86aedeb, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:04.400166] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 11:35:04.400338] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7fbfa8434490) allocated with standard calloc() >[2018-08-18 11:35:04.400805] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005 on vol0-replicate-32 >[2018-08-18 11:35:04.401065] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401091] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401116] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401135] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401145] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005 with op_ret -1 [No such file or directory] >[2018-08-18 11:35:04.401154] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005 missing on subvol vol0-replicate-32 >[2018-08-18 11:35:04.401162] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:04.401443] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401463] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401464] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401494] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401505] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-0 >[2018-08-18 11:35:04.401530] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401531] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401535] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401565] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401582] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401594] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401635] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-1 >[2018-08-18 11:35:04.401642] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401639] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401670] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401682] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-2 >[2018-08-18 11:35:04.401687] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401687] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401720] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401720] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401726] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401758] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401758] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-3 >[2018-08-18 11:35:04.401782] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401797] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-4 >[2018-08-18 11:35:04.401802] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401816] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401851] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401872] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401879] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401905] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401907] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-5 >[2018-08-18 11:35:04.401908] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401933] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401942] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401953] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401957] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-6 >[2018-08-18 11:35:04.401962] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.401988] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402002] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-7 >[2018-08-18 11:35:04.402011] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402031] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402083] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402084] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402096] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402110] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402096] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402124] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-8 >[2018-08-18 11:35:04.402152] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402157] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402154] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402168] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402186] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-9 >[2018-08-18 11:35:04.402189] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402201] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-10 >[2018-08-18 11:35:04.402218] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402243] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402247] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402264] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402274] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-11 >[2018-08-18 11:35:04.402301] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402316] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402359] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402374] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402374] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402375] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402406] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-12 >[2018-08-18 11:35:04.402417] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402436] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402436] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402453] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402457] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-13 >[2018-08-18 11:35:04.402470] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402484] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402497] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402492] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-14 >[2018-08-18 11:35:04.402508] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402517] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402537] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402548] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-15 >[2018-08-18 11:35:04.402551] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402592] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402631] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402629] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402668] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402672] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402669] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402701] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402711] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402715] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402713] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-17 >[2018-08-18 11:35:04.402724] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-16 >[2018-08-18 11:35:04.402718] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402756] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402779] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-18 >[2018-08-18 11:35:04.402808] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402831] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402867] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402876] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402891] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402903] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402920] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-19 >[2018-08-18 11:35:04.402903] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402941] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402954] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-20 >[2018-08-18 11:35:04.402957] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402954] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402956] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.402992] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403017] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403015] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-21 >[2018-08-18 11:35:04.403047] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403080] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403094] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403099] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403113] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-22 >[2018-08-18 11:35:04.403118] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403157] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403171] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403189] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-23 >[2018-08-18 11:35:04.403190] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403222] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403226] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403246] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403257] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-24 >[2018-08-18 11:35:04.403271] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403290] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403308] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403323] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403332] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-25 >[2018-08-18 11:35:04.403363] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403361] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403361] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403389] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403400] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-26 >[2018-08-18 11:35:04.403411] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403415] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403444] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403455] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403462] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-27 >[2018-08-18 11:35:04.403492] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403507] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403509] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403548] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403552] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403568] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-28 >[2018-08-18 11:35:04.403580] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403605] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403622] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403630] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-29 >[2018-08-18 11:35:04.403647] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403647] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403720] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403724] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403721] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403727] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403752] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403756] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403762] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-30 >[2018-08-18 11:35:04.403775] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-31 >[2018-08-18 11:35:04.403805] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403807] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403837] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403842] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403861] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403864] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403887] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403890] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403900] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-32 >[2018-08-18 11:35:04.403903] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-33 >[2018-08-18 11:35:04.403916] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403924] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403933] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403945] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403957] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-34 >[2018-08-18 11:35:04.403964] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.403986] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.404037] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.404051] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.404058] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-35 >[2018-08-18 11:35:04.404066] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-32 cached_subvol null >[2018-08-18 11:35:04.404074] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005 >[2018-08-18 11:35:04.404081] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x2147ae0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.404088] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.404096] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.404106] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.404115] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x2147ae0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.404732] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m returned with op_ret 0 >[2018-08-18 11:35:04.408147] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.408168] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m returned with op_ret 0" repeated 35 times between [2018-08-18 11:35:04.404732] and [2018-08-18 11:35:04.407837] >[2018-08-18 11:35:04.408167] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=165daf77-45ae-451a-b5a4-415ab86aedeb, gen=0): picking the request for winding >[2018-08-18 11:35:04.408206] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88 at vol0-replicate-25 >[2018-08-18 11:35:04.408385] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=165daf77-45ae-451a-b5a4-415ab86aedeb, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.408496] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88 returned with op_ret 0 >[2018-08-18 11:35:04.408602] D [md-cache.c:503:mdc_inode_iatt_set_validate] (-->/usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3cd0) [0x7fc07374dcd0] -->/usr/lib64/glusterfs/3.10.12/xlator/performance/md-cache.so(+0xa802) [0x7fc073334802] -->/usr/lib64/glusterfs/3.10.12/xlator/performance/md-cache.so(+0x66cb) [0x7fc0733306cb] ) 0-vol0-md-cache: discarding the iatt validate request(ctime_nsec) [Invalid argument] >[2018-08-18 11:35:04.408721] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005 on vol0-replicate-32 >[2018-08-18 11:35:04.408953] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.408959] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.408989] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409008] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409017] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005 with op_ret -1 [No such file or directory] >[2018-08-18 11:35:04.409025] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005 missing on subvol vol0-replicate-32 >[2018-08-18 11:35:04.409032] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:04.409257] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409308] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409313] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409339] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409339] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409370] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409373] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409389] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-0 >[2018-08-18 11:35:04.409403] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409405] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409407] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409431] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409452] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-1 >[2018-08-18 11:35:04.409454] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409455] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409455] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409483] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-2 >[2018-08-18 11:35:04.409491] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409493] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409509] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409514] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409524] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-3 >[2018-08-18 11:35:04.409514] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409546] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409552] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409555] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-4 >[2018-08-18 11:35:04.409574] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409604] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409609] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409629] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409631] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409642] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-5 >[2018-08-18 11:35:04.409636] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409658] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409667] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-6 >[2018-08-18 11:35:04.409700] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409706] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409707] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409743] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409752] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-7 >[2018-08-18 11:35:04.409754] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409778] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409802] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409813] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409821] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-8 >[2018-08-18 11:35:04.409829] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409831] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409835] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409860] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409875] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-9 >[2018-08-18 11:35:04.409908] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409915] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409936] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409939] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409960] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409964] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.409973] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-10 >[2018-08-18 11:35:04.409996] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410009] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410013] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410017] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-11 >[2018-08-18 11:35:04.410053] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410071] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410086] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410095] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-12 >[2018-08-18 11:35:04.410121] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410140] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410159] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410174] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410178] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410184] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-13 >[2018-08-18 11:35:04.410198] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410225] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410249] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410259] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-14 >[2018-08-18 11:35:04.410265] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410288] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410306] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410319] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410319] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410333] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-15 >[2018-08-18 11:35:04.410340] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410351] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410365] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410374] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-16 >[2018-08-18 11:35:04.410395] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410404] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410418] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410431] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410438] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410438] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-17 >[2018-08-18 11:35:04.410455] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410484] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410490] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410501] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410512] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410523] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-18 >[2018-08-18 11:35:04.410531] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410552] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410565] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410568] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-19 >[2018-08-18 11:35:04.410585] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410603] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410604] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410627] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410633] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410637] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-20 >[2018-08-18 11:35:04.410638] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410662] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410665] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410682] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410689] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-21 >[2018-08-18 11:35:04.410692] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410713] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410727] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-22 >[2018-08-18 11:35:04.410737] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410762] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410758] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410789] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410804] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-23 >[2018-08-18 11:35:04.410806] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410812] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410822] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410839] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410852] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410853] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-24 >[2018-08-18 11:35:04.410875] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410904] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410917] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410924] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-25 >[2018-08-18 11:35:04.410940] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410941] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410954] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410947] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410977] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.410986] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-26 >[2018-08-18 11:35:04.411005] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411023] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411056] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411073] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411093] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411100] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411107] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411121] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-27 >[2018-08-18 11:35:04.411129] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411151] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411157] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411160] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411171] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411176] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411201] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-29 >[2018-08-18 11:35:04.411205] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411202] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411223] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-28 >[2018-08-18 11:35:04.411228] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-30 >[2018-08-18 11:35:04.411229] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411263] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411265] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411269] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411274] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411280] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411283] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411321] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411324] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411331] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411338] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-32 >[2018-08-18 11:35:04.411337] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411355] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-31 >[2018-08-18 11:35:04.411366] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411364] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411383] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411397] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411398] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411406] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411412] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-34 >[2018-08-18 11:35:04.411408] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-33 >[2018-08-18 11:35:04.411420] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411481] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411494] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411501] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005) from subvol vol0-replicate-35 >[2018-08-18 11:35:04.411513] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-32 cached_subvol null >[2018-08-18 11:35:04.411521] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005 >[2018-08-18 11:35:04.411527] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411535] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411543] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411551] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411558] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00396b0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.411662] I [MSGID: 109066] [dht-rename.c:1610:dht_rename] 0-vol0-dht: renaming /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005_bfccc3c0a2da11e8b1bd6c92bf5c1b88 (hash=vol0-replicate-25/cache=vol0-replicate-25) => /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005 (hash=vol0-replicate-32/cache=<nul>) >[2018-08-18 11:35:04.412297] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-32: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.432340] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 11:35:04.432463] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-25: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.432789] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 11:35:04.432905] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-32: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.433403] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 11:35:04.433498] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-25: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.433806] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 11:35:04.433876] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-32: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.434074] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 11:35:04.434532] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.434552] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=165daf77-45ae-451a-b5a4-415ab86aedeb, gen=0): picking the request for winding >[2018-08-18 11:35:04.434577] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005 at vol0-replicate-25 >[2018-08-18 11:35:04.434774] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=165daf77-45ae-451a-b5a4-415ab86aedeb, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.434992] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005 returned with op_ret 0 >[2018-08-18 11:35:04.435275] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-25: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.435500] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 11:35:04.435675] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.435695] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=165daf77-45ae-451a-b5a4-415ab86aedeb, gen=0): picking the request for winding >[2018-08-18 11:35:04.435717] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005 at vol0-replicate-25 >[2018-08-18 11:35:04.435892] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=165daf77-45ae-451a-b5a4-415ab86aedeb, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.436053] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005 returned with op_ret 0 >[2018-08-18 11:35:04.436300] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-25: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.436499] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 11:35:04.436666] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.436687] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=165daf77-45ae-451a-b5a4-415ab86aedeb, gen=0): picking the request for winding >[2018-08-18 11:35:04.436708] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005 at vol0-replicate-25 >[2018-08-18 11:35:04.436883] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=165daf77-45ae-451a-b5a4-415ab86aedeb, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.437006] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005 returned with op_ret 0 >[2018-08-18 11:35:04.437231] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-25: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.437436] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 11:35:04.437601] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.437619] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=165daf77-45ae-451a-b5a4-415ab86aedeb, gen=0): picking the request for winding >[2018-08-18 11:35:04.437641] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005 at vol0-replicate-25 >[2018-08-18 11:35:04.437813] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=165daf77-45ae-451a-b5a4-415ab86aedeb, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.437932] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/65078a13650f3366b3b55ac8f14230ad_171654948218480833.m/0005 returned with op_ret 0 >[2018-08-18 11:35:04.438158] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-25: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.438355] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 11:35:04.443048] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7fbfa802ae80) as it exceeds the maximum available buffer size >[2018-08-18 11:35:04.443908] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.443927] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=e40b648a-8f88-4808-9583-9143d5a4444f, gen=0): picking the request for winding >[2018-08-18 11:35:04.443938] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=e40b648a-8f88-4808-9583-9143d5a4444f, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:04.443978] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.443989] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:04.444008] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7fbfa000d030) allocated with standard calloc() >[2018-08-18 11:35:04.456232] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.456256] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=1, gfid=e40b648a-8f88-4808-9583-9143d5a4444f. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=e40b648a-8f88-4808-9583-9143d5a4444f, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:04.460349] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=e40b648a-8f88-4808-9583-9143d5a4444f, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:04.460464] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=e40b648a-8f88-4808-9583-9143d5a4444f, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.460566] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.460585] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=e40b648a-8f88-4808-9583-9143d5a4444f, gen=1): picking the request for winding >[2018-08-18 11:35:04.460802] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=e40b648a-8f88-4808-9583-9143d5a4444f, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:04.460914] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:04.461098] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7fbfa802ae80) allocated with standard calloc() >[2018-08-18 11:35:04.461755] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004 on vol0-replicate-27 >[2018-08-18 11:35:04.462020] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462040] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462065] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462085] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462094] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004 with op_ret -1 [No such file or directory] >[2018-08-18 11:35:04.462102] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004 missing on subvol vol0-replicate-27 >[2018-08-18 11:35:04.462109] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:04.462318] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462344] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462376] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462410] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462418] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462432] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462430] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-0 >[2018-08-18 11:35:04.462466] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462491] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462493] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462500] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462530] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462538] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-1 >[2018-08-18 11:35:04.462557] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462555] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462556] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462583] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-2 >[2018-08-18 11:35:04.462587] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462627] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462636] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462646] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462644] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-3 >[2018-08-18 11:35:04.462636] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462691] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462693] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462694] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462707] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462714] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-4 >[2018-08-18 11:35:04.462726] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462741] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-5 >[2018-08-18 11:35:04.462767] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462789] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462814] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462828] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462837] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-6 >[2018-08-18 11:35:04.462864] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462889] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462915] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462928] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462936] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-7 >[2018-08-18 11:35:04.462944] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462966] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462982] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.462999] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463009] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-8 >[2018-08-18 11:35:04.463010] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463043] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463041] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463066] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463078] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463080] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-9 >[2018-08-18 11:35:04.463096] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463097] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463102] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463115] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463127] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463130] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463142] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-10 >[2018-08-18 11:35:04.463148] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463158] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-11 >[2018-08-18 11:35:04.463191] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463203] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463213] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463228] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463235] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-12 >[2018-08-18 11:35:04.463248] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463260] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463293] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463308] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463315] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-13 >[2018-08-18 11:35:04.463334] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463348] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463354] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463368] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463377] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-14 >[2018-08-18 11:35:04.463415] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463424] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463452] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463475] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463475] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463484] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-15 >[2018-08-18 11:35:04.463500] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463507] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463523] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463525] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463535] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-16 >[2018-08-18 11:35:04.463563] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463569] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463584] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463592] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-17 >[2018-08-18 11:35:04.463628] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463631] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463629] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463661] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463670] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-18 >[2018-08-18 11:35:04.463680] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463670] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463718] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463730] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463737] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-19 >[2018-08-18 11:35:04.463752] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463768] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463777] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463802] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463809] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-20 >[2018-08-18 11:35:04.463824] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463837] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463856] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463865] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463872] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-21 >[2018-08-18 11:35:04.463926] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463927] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463948] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463961] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463975] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-22 >[2018-08-18 11:35:04.463984] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.463987] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464002] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464016] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464024] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-23 >[2018-08-18 11:35:04.464034] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464060] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464054] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464082] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464096] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-24 >[2018-08-18 11:35:04.464111] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464125] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464126] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464147] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464157] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-25 >[2018-08-18 11:35:04.464158] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464170] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464197] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464201] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464211] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464216] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464220] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-26 >[2018-08-18 11:35:04.464272] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464277] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464316] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464333] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464333] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464342] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-28 >[2018-08-18 11:35:04.464352] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464374] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464390] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464399] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-29 >[2018-08-18 11:35:04.464409] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464427] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464429] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464448] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464456] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-30 >[2018-08-18 11:35:04.464465] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464486] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464508] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464508] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464526] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464535] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-31 >[2018-08-18 11:35:04.464538] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464558] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464572] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464580] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-32 >[2018-08-18 11:35:04.464611] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464630] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464635] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464650] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464670] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464657] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464679] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-33 >[2018-08-18 11:35:04.464693] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464699] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-27 >[2018-08-18 11:35:04.464706] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464722] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464738] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464739] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464767] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-34 >[2018-08-18 11:35:04.464777] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464791] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464808] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464815] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-35 >[2018-08-18 11:35:04.464824] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-27 cached_subvol null >[2018-08-18 11:35:04.464831] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004 >[2018-08-18 11:35:04.464838] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x2029970, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464845] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464853] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464861] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.464869] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.465167] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t on vol0-replicate-24 >[2018-08-18 11:35:04.465442] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m returned with op_ret 0 >[2018-08-18 11:35:04.465540] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t with op_ret 0 >[2018-08-18 11:35:04.465567] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-24 >[2018-08-18 11:35:04.465825] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: 6135251d-bd90-46da-a007-650da6fbe9d0: generation now vs cached: 3, 3 >[2018-08-18 11:35:04.466511] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.466531] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:04.466543] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:04.466759] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.467341] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:04.467453] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.467552] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.468392] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.468414] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m returned with op_ret 0" repeated 35 times between [2018-08-18 11:35:04.465442] and [2018-08-18 11:35:04.468107] >[2018-08-18 11:35:04.468413] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=e40b648a-8f88-4808-9583-9143d5a4444f, gen=0): picking the request for winding >[2018-08-18 11:35:04.468448] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 at vol0-replicate-10 >[2018-08-18 11:35:04.468637] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=e40b648a-8f88-4808-9583-9143d5a4444f, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.468723] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 returned with op_ret 0 >[2018-08-18 11:35:04.468823] D [md-cache.c:503:mdc_inode_iatt_set_validate] (-->/usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3cd0) [0x7fc07374dcd0] -->/usr/lib64/glusterfs/3.10.12/xlator/performance/md-cache.so(+0xa802) [0x7fc073334802] -->/usr/lib64/glusterfs/3.10.12/xlator/performance/md-cache.so(+0x66cb) [0x7fc0733306cb] ) 0-vol0-md-cache: discarding the iatt validate request(ctime_nsec) [Invalid argument] >[2018-08-18 11:35:04.468829] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.468850] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:04.468863] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:04.468906] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.468917] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:04.468935] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004 on vol0-replicate-27 >[2018-08-18 11:35:04.469153] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469170] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469172] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469198] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469206] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004 with op_ret -1 [No such file or directory] >[2018-08-18 11:35:04.469214] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004 missing on subvol vol0-replicate-27 >[2018-08-18 11:35:04.469220] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:04.469449] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469468] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469510] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469532] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469541] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-0 >[2018-08-18 11:35:04.469572] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469583] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469593] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469633] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469639] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469670] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-1 >[2018-08-18 11:35:04.469679] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469674] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469711] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469714] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469716] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469712] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469770] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469770] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-2 >[2018-08-18 11:35:04.469784] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-3 >[2018-08-18 11:35:04.469786] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469796] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.469807] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469817] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:04.469811] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469830] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:04.469860] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469858] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469858] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469879] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469891] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-4 >[2018-08-18 11:35:04.469917] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469921] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469925] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469929] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-5 >[2018-08-18 11:35:04.469967] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469969] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469990] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.469998] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470008] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470002] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470019] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470030] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-7 >[2018-08-18 11:35:04.470031] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470030] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-6 >[2018-08-18 11:35:04.470030] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470067] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470075] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-8 >[2018-08-18 11:35:04.470080] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470103] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.470099] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470101] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470133] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470134] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470148] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470149] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-9 >[2018-08-18 11:35:04.470167] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470186] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470195] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-10 >[2018-08-18 11:35:04.470195] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470208] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470211] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 11:35:04.470230] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:04.470248] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470266] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470267] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470281] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-11 >[2018-08-18 11:35:04.470283] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470284] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470311] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470319] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-12 >[2018-08-18 11:35:04.470329] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:04.470350] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470362] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470376] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470396] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470403] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-13 >[2018-08-18 11:35:04.470428] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470442] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470445] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470466] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470479] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-14 >[2018-08-18 11:35:04.470492] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470498] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470533] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.470549] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): picking the request for winding >[2018-08-18 11:35:04.470550] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470561] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:04.470566] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470570] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:04.470567] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470567] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470570] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470613] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.470576] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-15 >[2018-08-18 11:35:04.470628] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470625] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470649] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470647] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-16 >[2018-08-18 11:35:04.470653] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470647] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470656] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470669] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470680] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470675] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470689] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-18 >[2018-08-18 11:35:04.470699] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-17 >[2018-08-18 11:35:04.470710] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470715] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470739] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470753] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470762] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-19 >[2018-08-18 11:35:04.470780] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470786] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470806] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470817] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470822] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.470816] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470832] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470835] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-20 >[2018-08-18 11:35:04.470840] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:04.470851] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:04.470851] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470859] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:04.470864] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470885] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-21 >[2018-08-18 11:35:04.470884] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470892] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470891] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470911] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470920] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-22 >[2018-08-18 11:35:04.470925] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470942] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470949] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470959] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.470966] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-23 >[2018-08-18 11:35:04.470998] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471004] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471036] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471052] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471053] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471064] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-24 >[2018-08-18 11:35:04.471070] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.471080] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471081] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471088] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:04.471099] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:04.471096] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471097] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471108] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:04.471114] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-25 >[2018-08-18 11:35:04.471150] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471153] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471177] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471199] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471216] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471221] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471225] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-26 >[2018-08-18 11:35:04.471251] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471256] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471262] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471257] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471287] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471297] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471298] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471293] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471306] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471293] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471322] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471323] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471324] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471331] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471345] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471350] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471350] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-30 >[2018-08-18 11:35:04.471354] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471381] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-28 >[2018-08-18 11:35:04.471345] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-31 >[2018-08-18 11:35:04.471395] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471357] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-29 >[2018-08-18 11:35:04.471396] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471417] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471420] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471421] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471431] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471434] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-32 >[2018-08-18 11:35:04.471423] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471442] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471449] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471444] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-27 >[2018-08-18 11:35:04.471457] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471461] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471478] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471474] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471475] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-33 >[2018-08-18 11:35:04.471503] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-34 >[2018-08-18 11:35:04.471507] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471513] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471525] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471532] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004) from subvol vol0-replicate-35 >[2018-08-18 11:35:04.471540] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-27 cached_subvol null >[2018-08-18 11:35:04.471547] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004 >[2018-08-18 11:35:04.471553] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471560] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471567] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471575] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471582] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.471686] I [MSGID: 109066] [dht-rename.c:1610:dht_rename] 0-vol0-dht: renaming /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004_bfab2d1ea2da11e8a3196c92bf5c1b88 (hash=vol0-replicate-10/cache=vol0-replicate-10) => /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004 (hash=vol0-replicate-27/cache=<nul>) >[2018-08-18 11:35:04.471723] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:04.471831] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.471933] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.471947] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): picking the request for winding >[2018-08-18 11:35:04.471957] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): picking the request for winding >[2018-08-18 11:35:04.471968] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:04.472004] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.472015] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:04.472091] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.472294] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-27: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.472295] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:04.472395] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:04.472491] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.472507] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:04.472546] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:04.472676] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:04.472768] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:04.472856] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.472870] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:04.472914] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:04.473023] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:04.473115] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:04.473201] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.473213] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): picking the request for winding >[2018-08-18 11:35:04.473378] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): destroying request, removing from all queues >[2018-08-18 11:35:04.473465] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:04.484117] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-27: lk op is for a transaction >[2018-08-18 11:35:04.484214] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-10: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.484501] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:04.484590] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-27: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.485040] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-27: lk op is for a transaction >[2018-08-18 11:35:04.485128] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-10: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:04.485400] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:04.485452] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-27: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.485628] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-27: lk op is for a transaction >[2018-08-18 11:35:04.486120] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.486141] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=e40b648a-8f88-4808-9583-9143d5a4444f, gen=0): picking the request for winding >[2018-08-18 11:35:04.486162] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004 at vol0-replicate-10 >[2018-08-18 11:35:04.486353] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=e40b648a-8f88-4808-9583-9143d5a4444f, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.486440] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004 returned with op_ret 0 >[2018-08-18 11:35:04.486664] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.486860] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:04.487031] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.487047] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=e40b648a-8f88-4808-9583-9143d5a4444f, gen=0): picking the request for winding >[2018-08-18 11:35:04.487069] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004 at vol0-replicate-10 >[2018-08-18 11:35:04.487241] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=e40b648a-8f88-4808-9583-9143d5a4444f, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.487360] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004 returned with op_ret 0 >[2018-08-18 11:35:04.487571] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.487750] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:04.487919] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.487938] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=e40b648a-8f88-4808-9583-9143d5a4444f, gen=0): picking the request for winding >[2018-08-18 11:35:04.487960] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004 at vol0-replicate-10 >[2018-08-18 11:35:04.488132] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=e40b648a-8f88-4808-9583-9143d5a4444f, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.488230] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004 returned with op_ret 0 >[2018-08-18 11:35:04.488424] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.488588] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:04.488760] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:04.488779] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=e40b648a-8f88-4808-9583-9143d5a4444f, gen=0): picking the request for winding >[2018-08-18 11:35:04.488800] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004 at vol0-replicate-10 >[2018-08-18 11:35:04.488968] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=e40b648a-8f88-4808-9583-9143d5a4444f, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:04.489065] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/0760dee6406533f5aefa43f83bdd8918_171654947375444628.m/0004 returned with op_ret 0 >[2018-08-18 11:35:04.489268] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:04.489446] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:04.561331] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-6 >[2018-08-18 11:35:04.561744] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m with op_ret 0 >[2018-08-18 11:35:04.562000] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562011] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562044] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562064] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562081] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-0 returned error [No such file or directory] >[2018-08-18 11:35:04.562161] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562219] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562219] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562283] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562286] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562289] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562322] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562294] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562323] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562353] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562360] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-2 returned error [No such file or directory] >[2018-08-18 11:35:04.562419] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562421] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562423] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562446] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562440] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-3 returned error [No such file or directory] >[2018-08-18 11:35:04.562462] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562466] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-4 returned error [No such file or directory] >[2018-08-18 11:35:04.562479] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562489] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-5 returned error [No such file or directory] >[2018-08-18 11:35:04.562511] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562617] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562636] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562682] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562706] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562725] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562727] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562746] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-9 returned error [No such file or directory] >[2018-08-18 11:35:04.562754] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562778] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562791] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562798] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-10 returned error [No such file or directory] >[2018-08-18 11:35:04.562831] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562838] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562856] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562866] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562873] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-11 returned error [No such file or directory] >[2018-08-18 11:35:04.562899] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562941] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562965] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.562997] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563002] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563024] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563052] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-12 returned error [No such file or directory] >[2018-08-18 11:35:04.563057] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563081] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563090] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563099] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563154] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563169] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563174] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563171] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563174] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-14 returned error [No such file or directory] >[2018-08-18 11:35:04.563217] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563227] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-13 returned error [No such file or directory] >[2018-08-18 11:35:04.563239] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563272] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-15 returned error [No such file or directory] >[2018-08-18 11:35:04.563285] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563292] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563331] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563337] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563356] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563359] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563394] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563395] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-17 returned error [No such file or directory] >[2018-08-18 11:35:04.563400] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563446] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563450] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563447] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563474] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-18 returned error [No such file or directory] >[2018-08-18 11:35:04.563474] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563497] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563522] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-19 returned error [No such file or directory] >[2018-08-18 11:35:04.563541] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563541] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563581] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563615] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563636] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-21 returned error [No such file or directory] >[2018-08-18 11:35:04.563655] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563689] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563798] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563807] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563813] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563817] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563840] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-23 returned error [No such file or directory] >[2018-08-18 11:35:04.563852] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.563978] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564076] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564090] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564135] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564146] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564156] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564167] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564175] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564191] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-28 returned error [No such file or directory] >[2018-08-18 11:35:04.564258] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564258] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564297] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564330] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564353] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-29 returned error [No such file or directory] >[2018-08-18 11:35:04.564378] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564409] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564414] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9000e660, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564425] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-30 returned error [No such file or directory] >[2018-08-18 11:35:04.564447] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9000e660, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564482] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564486] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564502] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564491] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564526] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564542] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-32 returned error [No such file or directory] >[2018-08-18 11:35:04.564537] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564594] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564612] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564595] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c009700, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564636] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564658] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c009700, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564657] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-33 returned error [No such file or directory] >[2018-08-18 11:35:04.564671] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564699] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564711] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564717] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564722] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564743] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564741] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.564756] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-35 returned error [No such file or directory] >[2018-08-18 11:35:04.564772] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-34 returned error [No such file or directory] >[2018-08-18 11:35:04.565086] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-21: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.565085] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-75: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.565109] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9000e660, vol0-client-21 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.565119] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c009700, vol0-client-75 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.565130] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-22: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.565122] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-76: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.565142] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-23: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.565141] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9000e660, vol0-client-22 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.565164] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9000e660, vol0-client-23 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.565163] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-77: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.565163] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c009700, vol0-client-76 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.565180] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c009700, vol0-client-77 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.566331] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9801bed0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.566360] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c013160, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.566803] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-61: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.566818] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9801bed0, vol0-client-61 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.566823] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-60: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.566835] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-73: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.566856] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c013160, vol0-client-73 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.566859] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-72: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.566857] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-62: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.566875] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c013160, vol0-client-72 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.566872] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9801bed0, vol0-client-60 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.566882] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9801bed0, vol0-client-62 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.566885] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-74: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.566920] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9c013160, vol0-client-74 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.568970] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-6 >[2018-08-18 11:35:04.569368] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m with op_ret 0 >[2018-08-18 11:35:04.569658] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.569728] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.569728] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.569770] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.569783] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-0 returned error [No such file or directory] >[2018-08-18 11:35:04.569902] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.569919] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.569953] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.569965] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.569974] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-2 returned error [No such file or directory] >[2018-08-18 11:35:04.570038] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.570069] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.570099] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.570110] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.570118] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-3 returned error [No such file or directory] >[2018-08-18 11:35:04.570146] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.570179] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.570189] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.570210] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.570219] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-4 returned error [No such file or directory] >[2018-08-18 11:35:04.570466] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.570488] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.570686] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.570718] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.570719] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.570755] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.570760] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.570776] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-9 returned error [No such file or directory] >[2018-08-18 11:35:04.570794] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.570820] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.570838] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.570847] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-10 returned error [No such file or directory] >[2018-08-18 11:35:04.570924] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.570947] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571002] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571096] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571134] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571144] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571179] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571231] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571253] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571265] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571276] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-14 returned error [No such file or directory] >[2018-08-18 11:35:04.571304] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571324] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571343] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571353] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571361] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-15 returned error [No such file or directory] >[2018-08-18 11:35:04.571483] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571503] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571521] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571530] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571537] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-17 returned error [No such file or directory] >[2018-08-18 11:35:04.571565] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571590] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571620] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571633] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571646] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-18 returned error [No such file or directory] >[2018-08-18 11:35:04.571673] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571690] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571706] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571715] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.571721] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-19 returned error [No such file or directory] >[2018-08-18 11:35:04.571774] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.572261] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.572946] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.572996] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.573066] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.573262] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.573302] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.573735] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49222: ping timer event already removed >[2018-08-18 11:35:04.574067] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf80005bc0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.574090] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49222: ping timer event already removed >[2018-08-18 11:35:04.574173] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.574193] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf80005bc0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.574209] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.574604] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49222: ping timer event already removed >[2018-08-18 11:35:04.574733] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.574769] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.574793] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.574809] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.574822] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-29 returned error [No such file or directory] >[2018-08-18 11:35:04.574877] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.574993] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.575038] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf7800bcb0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.575631] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.575664] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf7800bcb0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.575948] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.575985] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c005090, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.576012] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.576034] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.576044] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.576052] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on vol0-replicate-33 returned error [No such file or directory] >[2018-08-18 11:35:04.576075] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.576099] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf84005fa0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.576144] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf84005fa0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.576159] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.576291] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-21: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.576303] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf80005bc0, vol0-client-21 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.576321] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-22: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.576329] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf80005bc0, vol0-client-22 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.576343] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-23: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.576351] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf80005bc0, vol0-client-23 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.576403] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf90015f70, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.576463] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c00f650, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.576484] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf6c00f650, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.576504] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-37: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.576508] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-36: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.576514] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf7800bcb0, vol0-client-37 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.576528] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf7800bcb0, vol0-client-36 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.576546] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf980b6c80, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.576542] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-38: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.576550] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf980b6c80, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.576564] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf7800bcb0, vol0-client-38 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.576658] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf80003240, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.576667] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf80003240, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.576919] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-96: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.576939] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf6c00f650, vol0-client-96 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.576945] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-97: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.576957] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-98: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.576998] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf6c00f650, vol0-client-98 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.577002] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf6c00f650, vol0-client-97 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.577066] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf80014930, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.577072] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-39: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.577067] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-40: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.577096] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-41: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.577115] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf6c005090, vol0-client-39 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.577128] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf6c005090, vol0-client-41 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.577131] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf6c005090, vol0-client-40 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.577161] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-70: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.577175] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf80003240, vol0-client-70 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.577162] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-69: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.577189] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf80003240, vol0-client-69 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.577215] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-71: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.577229] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf80003240, vol0-client-71 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.577535] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-63: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.577540] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-64: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.577545] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-104: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.577543] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-103: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.577565] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf980b6c80, vol0-client-104 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.577572] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf980b6c80, vol0-client-103 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.577575] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf90015f70, vol0-client-64 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.577588] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf90015f70, vol0-client-63 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.577612] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-65: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.577610] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-102: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.577627] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf90015f70, vol0-client-65 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.577637] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf980b6c80, vol0-client-102 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.577663] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-33: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.577700] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf84005fa0, vol0-client-33 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.577705] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-35: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.577701] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-34: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.577729] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf84005fa0, vol0-client-34 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.577730] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf84005fa0, vol0-client-35 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.577854] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-73: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.577865] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf80014930, vol0-client-73 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.577859] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-72: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.577868] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-74: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.577881] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf80014930, vol0-client-72 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.577886] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf80014930, vol0-client-74 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.578396] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8001cb70, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.578619] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9000e1f0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.578636] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9000e1f0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:04.579151] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-75: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.579158] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-76: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.579169] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf8001cb70, vol0-client-75 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.579170] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-77: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.579184] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf8001cb70, vol0-client-76 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.579203] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf8001cb70, vol0-client-77 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.579365] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-84: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.579374] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9000e1f0, vol0-client-84 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.579375] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-85: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.579384] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-86: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:04.579392] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9000e1f0, vol0-client-85 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:04.579401] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7fbf9000e1f0, vol0-client-86 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.084261] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m on vol0-replicate-9 >[2018-08-18 11:35:05.084626] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49197: ping timer event already removed >[2018-08-18 11:35:05.084718] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.084838] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49197: ping timer event already removed >[2018-08-18 11:35:05.084909] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.085045] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49197: ping timer event already removed >[2018-08-18 11:35:05.085112] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.085146] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.085158] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m with op_ret -1 [No such file or directory] >[2018-08-18 11:35:05.085176] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m missing on subvol vol0-replicate-9 >[2018-08-18 11:35:05.085184] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:05.085476] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49188: ping timer event already removed >[2018-08-18 11:35:05.085533] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.085674] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49188: ping timer event already removed >[2018-08-18 11:35:05.085741] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.085844] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49188: ping timer event already removed >[2018-08-18 11:35:05.085920] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.085960] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.085974] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-0 >[2018-08-18 11:35:05.086037] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49189: ping timer event already removed >[2018-08-18 11:35:05.086112] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.086187] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49189: ping timer event already removed >[2018-08-18 11:35:05.086262] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.086345] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49189: ping timer event already removed >[2018-08-18 11:35:05.086441] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.086474] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.086489] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-1 >[2018-08-18 11:35:05.086527] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49190: ping timer event already removed >[2018-08-18 11:35:05.086622] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.086687] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49190: ping timer event already removed >[2018-08-18 11:35:05.086757] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.086840] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49190: ping timer event already removed >[2018-08-18 11:35:05.086933] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.086953] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.086968] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-2 >[2018-08-18 11:35:05.087026] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49191: ping timer event already removed >[2018-08-18 11:35:05.087100] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.087175] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49191: ping timer event already removed >[2018-08-18 11:35:05.087263] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.087328] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49191: ping timer event already removed >[2018-08-18 11:35:05.087395] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.087410] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.087417] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-3 >[2018-08-18 11:35:05.087507] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49192: ping timer event already removed >[2018-08-18 11:35:05.087592] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.087668] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49192: ping timer event already removed >[2018-08-18 11:35:05.087720] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.087829] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49192: ping timer event already removed >[2018-08-18 11:35:05.087890] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.087918] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.087930] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-4 >[2018-08-18 11:35:05.088005] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49193: ping timer event already removed >[2018-08-18 11:35:05.088091] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.088158] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49193: ping timer event already removed >[2018-08-18 11:35:05.088222] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.088308] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49193: ping timer event already removed >[2018-08-18 11:35:05.088391] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.088412] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.088423] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-5 >[2018-08-18 11:35:05.088480] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49194: ping timer event already removed >[2018-08-18 11:35:05.088617] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.088660] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49194: ping timer event already removed >[2018-08-18 11:35:05.088716] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.088825] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49194: ping timer event already removed >[2018-08-18 11:35:05.088909] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.088931] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.088944] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-6 >[2018-08-18 11:35:05.089009] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49195: ping timer event already removed >[2018-08-18 11:35:05.089095] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.089161] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49195: ping timer event already removed >[2018-08-18 11:35:05.089230] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.089317] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49195: ping timer event already removed >[2018-08-18 11:35:05.089380] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.089408] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.089422] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-7 >[2018-08-18 11:35:05.089498] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49196: ping timer event already removed >[2018-08-18 11:35:05.089560] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.089665] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49196: ping timer event already removed >[2018-08-18 11:35:05.089728] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.089821] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49196: ping timer event already removed >[2018-08-18 11:35:05.089888] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.089905] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.089913] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-8 >[2018-08-18 11:35:05.090026] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.090058] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.090083] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49198: ping timer event already removed >[2018-08-18 11:35:05.090129] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.090240] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49198: ping timer event already removed >[2018-08-18 11:35:05.090311] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.090391] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49198: ping timer event already removed >[2018-08-18 11:35:05.090448] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.090465] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.090483] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-10 >[2018-08-18 11:35:05.090568] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49199: ping timer event already removed >[2018-08-18 11:35:05.090649] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.090754] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49199: ping timer event already removed >[2018-08-18 11:35:05.090803] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.090908] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49199: ping timer event already removed >[2018-08-18 11:35:05.090966] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.090983] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.091004] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-11 >[2018-08-18 11:35:05.091087] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49200: ping timer event already removed >[2018-08-18 11:35:05.091163] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.091239] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49200: ping timer event already removed >[2018-08-18 11:35:05.091313] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.091389] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49200: ping timer event already removed >[2018-08-18 11:35:05.091474] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.091494] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.091504] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-12 >[2018-08-18 11:35:05.091566] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49201: ping timer event already removed >[2018-08-18 11:35:05.091656] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.091748] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49201: ping timer event already removed >[2018-08-18 11:35:05.091806] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.091933] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49201: ping timer event already removed >[2018-08-18 11:35:05.091992] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.092018] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.092032] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-13 >[2018-08-18 11:35:05.092168] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49202: ping timer event already removed >[2018-08-18 11:35:05.092226] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.092356] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49202: ping timer event already removed >[2018-08-18 11:35:05.092401] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.092512] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49202: ping timer event already removed >[2018-08-18 11:35:05.092616] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.092660] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.092681] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-14 >[2018-08-18 11:35:05.092695] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49203: ping timer event already removed >[2018-08-18 11:35:05.092760] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.092881] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49203: ping timer event already removed >[2018-08-18 11:35:05.092955] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.093030] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49203: ping timer event already removed >[2018-08-18 11:35:05.093133] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.093159] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.093174] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-15 >[2018-08-18 11:35:05.093248] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49204: ping timer event already removed >[2018-08-18 11:35:05.093300] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.093429] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49204: ping timer event already removed >[2018-08-18 11:35:05.093493] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.093628] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49204: ping timer event already removed >[2018-08-18 11:35:05.093678] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.093701] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.093711] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-16 >[2018-08-18 11:35:05.093806] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49205: ping timer event already removed >[2018-08-18 11:35:05.093882] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.093954] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49205: ping timer event already removed >[2018-08-18 11:35:05.094041] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.094106] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49205: ping timer event already removed >[2018-08-18 11:35:05.094170] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.094201] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.094214] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-17 >[2018-08-18 11:35:05.094290] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49206: ping timer event already removed >[2018-08-18 11:35:05.094348] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.094441] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49206: ping timer event already removed >[2018-08-18 11:35:05.094517] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.094589] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49206: ping timer event already removed >[2018-08-18 11:35:05.094696] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.094724] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.094739] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-18 >[2018-08-18 11:35:05.094829] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49207: ping timer event already removed >[2018-08-18 11:35:05.094882] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.094991] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49207: ping timer event already removed >[2018-08-18 11:35:05.095073] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.095151] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49207: ping timer event already removed >[2018-08-18 11:35:05.095236] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.095263] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.095278] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-19 >[2018-08-18 11:35:05.095364] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49208: ping timer event already removed >[2018-08-18 11:35:05.095415] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.095545] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49208: ping timer event already removed >[2018-08-18 11:35:05.095605] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.095727] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49208: ping timer event already removed >[2018-08-18 11:35:05.095792] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.095816] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.095848] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-20 >[2018-08-18 11:35:05.095944] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49209: ping timer event already removed >[2018-08-18 11:35:05.095996] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.096097] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49209: ping timer event already removed >[2018-08-18 11:35:05.096166] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.096241] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49209: ping timer event already removed >[2018-08-18 11:35:05.096322] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.096342] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.096353] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-21 >[2018-08-18 11:35:05.096413] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49210: ping timer event already removed >[2018-08-18 11:35:05.096476] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.096563] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49210: ping timer event already removed >[2018-08-18 11:35:05.096645] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.096716] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49210: ping timer event already removed >[2018-08-18 11:35:05.096783] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.096798] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.096808] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-22 >[2018-08-18 11:35:05.096913] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49211: ping timer event already removed >[2018-08-18 11:35:05.096981] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.097063] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49211: ping timer event already removed >[2018-08-18 11:35:05.097133] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.097207] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49211: ping timer event already removed >[2018-08-18 11:35:05.097268] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.097294] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.097302] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-23 >[2018-08-18 11:35:05.097449] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.097464] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49213: ping timer event already removed >[2018-08-18 11:35:05.097469] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.097494] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.097524] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.097536] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-24 >[2018-08-18 11:35:05.097538] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.097616] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49213: ping timer event already removed >[2018-08-18 11:35:05.097692] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.097765] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49213: ping timer event already removed >[2018-08-18 11:35:05.097841] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.097858] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.097887] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-25 >[2018-08-18 11:35:05.097943] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49214: ping timer event already removed >[2018-08-18 11:35:05.098035] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.098121] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49214: ping timer event already removed >[2018-08-18 11:35:05.098185] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.098274] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49214: ping timer event already removed >[2018-08-18 11:35:05.098354] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.098370] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.098380] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-26 >[2018-08-18 11:35:05.098442] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49215: ping timer event already removed >[2018-08-18 11:35:05.098502] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.098624] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49215: ping timer event already removed >[2018-08-18 11:35:05.098675] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.098793] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49215: ping timer event already removed >[2018-08-18 11:35:05.098852] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.098872] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.098893] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-27 >[2018-08-18 11:35:05.098969] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49216: ping timer event already removed >[2018-08-18 11:35:05.099044] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.099112] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49216: ping timer event already removed >[2018-08-18 11:35:05.099210] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.099275] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49216: ping timer event already removed >[2018-08-18 11:35:05.099322] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.099337] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.099354] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-28 >[2018-08-18 11:35:05.099459] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49217: ping timer event already removed >[2018-08-18 11:35:05.099516] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.099609] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49217: ping timer event already removed >[2018-08-18 11:35:05.099693] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.099761] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49217: ping timer event already removed >[2018-08-18 11:35:05.099814] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.099830] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.099840] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-29 >[2018-08-18 11:35:05.099944] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49218: ping timer event already removed >[2018-08-18 11:35:05.099999] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.100096] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49218: ping timer event already removed >[2018-08-18 11:35:05.100170] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.100243] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49218: ping timer event already removed >[2018-08-18 11:35:05.100309] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.100337] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.100348] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-30 >[2018-08-18 11:35:05.100417] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49219: ping timer event already removed >[2018-08-18 11:35:05.100493] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.100562] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49219: ping timer event already removed >[2018-08-18 11:35:05.100636] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.100719] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49219: ping timer event already removed >[2018-08-18 11:35:05.100795] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.100835] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.100850] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-31 >[2018-08-18 11:35:05.100935] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49220: ping timer event already removed >[2018-08-18 11:35:05.100982] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.101093] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49220: ping timer event already removed >[2018-08-18 11:35:05.101184] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.101272] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49220: ping timer event already removed >[2018-08-18 11:35:05.101320] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.101334] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.101342] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-32 >[2018-08-18 11:35:05.101453] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49221: ping timer event already removed >[2018-08-18 11:35:05.101561] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.101623] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49221: ping timer event already removed >[2018-08-18 11:35:05.101668] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.101786] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49221: ping timer event already removed >[2018-08-18 11:35:05.101859] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.101886] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.101901] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-33 >[2018-08-18 11:35:05.101975] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49222: ping timer event already removed >[2018-08-18 11:35:05.102047] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.102131] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49222: ping timer event already removed >[2018-08-18 11:35:05.102200] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.102293] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49222: ping timer event already removed >[2018-08-18 11:35:05.102352] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.102370] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.102380] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-34 >[2018-08-18 11:35:05.102464] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49223: ping timer event already removed >[2018-08-18 11:35:05.102549] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.102624] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49223: ping timer event already removed >[2018-08-18 11:35:05.102685] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.102775] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49223: ping timer event already removed >[2018-08-18 11:35:05.102841] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.102849] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.102858] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.102872] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-35 >[2018-08-18 11:35:05.102874] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.102885] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-9 >[2018-08-18 11:35:05.102897] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-9 cached_subvol null >[2018-08-18 11:35:05.102904] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m >[2018-08-18 11:35:05.102911] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.102920] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.102930] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.102944] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.102953] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.103453] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0 returned with op_ret 0 >[2018-08-18 11:35:05.106686] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0 returned with op_ret 0" repeated 35 times between [2018-08-18 11:35:05.103453] and [2018-08-18 11:35:05.106421] >[2018-08-18 11:35:05.106684] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m on vol0-replicate-9 >[2018-08-18 11:35:05.106911] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.106943] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.106945] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.106973] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.106983] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m with op_ret -1 [No such file or directory] >[2018-08-18 11:35:05.106991] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m missing on subvol vol0-replicate-9 >[2018-08-18 11:35:05.106998] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:05.107228] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107235] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107253] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107279] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107285] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107290] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107296] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-0 >[2018-08-18 11:35:05.107302] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107335] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107342] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107349] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-1 >[2018-08-18 11:35:05.107343] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107362] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107384] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107394] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107414] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-2 >[2018-08-18 11:35:05.107421] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107424] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107454] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107457] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107467] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-3 >[2018-08-18 11:35:05.107458] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107474] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107490] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107507] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-4 >[2018-08-18 11:35:05.107520] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107541] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107559] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107575] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107585] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-5 >[2018-08-18 11:35:05.107622] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107621] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107666] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107663] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107683] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107630] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107715] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107724] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107732] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107720] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107734] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-7 >[2018-08-18 11:35:05.107726] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107752] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-6 >[2018-08-18 11:35:05.107783] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107788] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107804] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-8 >[2018-08-18 11:35:05.107805] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107868] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107900] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107900] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107928] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.107941] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-10 >[2018-08-18 11:35:05.107959] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108007] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108011] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108022] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108034] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108044] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108041] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108051] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-11 >[2018-08-18 11:35:05.108083] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108079] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108113] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108132] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-12 >[2018-08-18 11:35:05.108134] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108151] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108161] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108161] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108187] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108189] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108203] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108213] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-13 >[2018-08-18 11:35:05.108206] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-14 >[2018-08-18 11:35:05.108221] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108249] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108269] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108282] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-15 >[2018-08-18 11:35:05.108294] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108315] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108345] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108368] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108382] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-16 >[2018-08-18 11:35:05.108423] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108429] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108429] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108456] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108467] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-17 >[2018-08-18 11:35:05.108491] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108531] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108535] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108537] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108570] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108577] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108585] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-18 >[2018-08-18 11:35:05.108615] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108632] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108641] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108652] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108653] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108663] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-19 >[2018-08-18 11:35:05.108664] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108673] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108677] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108687] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108695] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-20 >[2018-08-18 11:35:05.108710] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108728] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-21 >[2018-08-18 11:35:05.108729] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108735] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108751] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108776] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108790] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-22 >[2018-08-18 11:35:05.108794] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108814] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108822] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108849] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108857] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-23 >[2018-08-18 11:35:05.108893] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108919] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108920] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108938] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108948] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-24 >[2018-08-18 11:35:05.108961] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.108969] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109006] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109037] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109048] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109048] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109057] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-25 >[2018-08-18 11:35:05.109099] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109103] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109105] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109115] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109145] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109147] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109158] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109166] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-27 >[2018-08-18 11:35:05.109172] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109175] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109197] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109203] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-26 >[2018-08-18 11:35:05.109205] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109229] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109234] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109238] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-28 >[2018-08-18 11:35:05.109231] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109259] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109261] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109276] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-29 >[2018-08-18 11:35:05.109316] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109338] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109338] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109349] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109350] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-30 >[2018-08-18 11:35:05.109406] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109428] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109433] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109435] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109458] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109473] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109474] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-31 >[2018-08-18 11:35:05.109486] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109507] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109516] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109510] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-32 >[2018-08-18 11:35:05.109515] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109521] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109535] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109572] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109579] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-9 >[2018-08-18 11:35:05.109581] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109603] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109605] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109604] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109605] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109637] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-34 >[2018-08-18 11:35:05.109631] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109629] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-33 >[2018-08-18 11:35:05.109662] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109676] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m) from subvol vol0-replicate-35 >[2018-08-18 11:35:05.109685] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-9 cached_subvol null >[2018-08-18 11:35:05.109691] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m >[2018-08-18 11:35:05.109697] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109704] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109711] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109723] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109731] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf9c01d020, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.109992] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-0': avail_percent is: 96.00 and avail_space is: 7682550996992 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110015] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-1': avail_percent is: 95.00 and avail_space is: 7649209065472 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110059] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-2': avail_percent is: 96.00 and avail_space is: 7679628898304 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110089] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-3': avail_percent is: 95.00 and avail_space is: 7676713996288 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110126] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-4': avail_percent is: 96.00 and avail_space is: 7679852584960 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110153] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-5': avail_percent is: 95.00 and avail_space is: 7672041328640 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110175] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-6': avail_percent is: 94.00 and avail_space is: 7567700049920 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110219] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-7': avail_percent is: 95.00 and avail_space is: 7599561781248 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110249] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-8': avail_percent is: 95.00 and avail_space is: 7669932085248 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110344] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-9': avail_percent is: 95.00 and avail_space is: 7675205881856 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110365] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-10': avail_percent is: 94.00 and avail_space is: 7587254632448 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110401] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-12': avail_percent is: 95.00 and avail_space is: 7662336352256 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110454] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-11': avail_percent is: 95.00 and avail_space is: 7666188136448 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110472] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-14': avail_percent is: 95.00 and avail_space is: 7665784659968 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110494] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-13': avail_percent is: 95.00 and avail_space is: 7671819669504 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110535] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-16': avail_percent is: 95.00 and avail_space is: 7678806634496 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110561] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-15': avail_percent is: 95.00 and avail_space is: 7654391312384 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110595] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-17': avail_percent is: 95.00 and avail_space is: 7669472206848 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110636] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-18': avail_percent is: 95.00 and avail_space is: 7615159742464 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110678] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-19': avail_percent is: 95.00 and avail_space is: 7660364656640 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110730] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-20': avail_percent is: 95.00 and avail_space is: 7610263240704 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110749] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-21': avail_percent is: 95.00 and avail_space is: 7663397208064 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110782] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-22': avail_percent is: 95.00 and avail_space is: 7668074450944 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110826] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-23': avail_percent is: 95.00 and avail_space is: 7669677629440 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110885] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-24': avail_percent is: 95.00 and avail_space is: 7660807667712 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110919] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-25': avail_percent is: 95.00 and avail_space is: 7644131004416 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110963] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-26': avail_percent is: 95.00 and avail_space is: 7655957123072 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.110994] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-27': avail_percent is: 95.00 and avail_space is: 7652166332416 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.111058] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-29': avail_percent is: 95.00 and avail_space is: 7639356854272 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.111077] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-28': avail_percent is: 95.00 and avail_space is: 7652197793792 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.111112] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-30': avail_percent is: 95.00 and avail_space is: 7675628605440 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.111180] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-31': avail_percent is: 95.00 and avail_space is: 7678164856832 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.111205] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-32': avail_percent is: 95.00 and avail_space is: 7659078995968 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.111279] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-34': avail_percent is: 95.00 and avail_space is: 7665343188992 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.111298] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-33': avail_percent is: 95.00 and avail_space is: 7676538961920 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.111320] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-35': avail_percent is: 95.00 and avail_space is: 7661850447872 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.111587] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-9: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.112076] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-0: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.112107] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-1: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.112142] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-2: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.112171] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-3: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.112225] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-4: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.112262] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-5: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.112315] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-6: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.112344] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-7: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.112390] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-8: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.112444] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-10: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.112481] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 11:35:05.112506] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-11: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.112541] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-12: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.112626] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-13: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.112681] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-14: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.112694] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 11:35:05.112771] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-15: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.112834] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-16: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.112910] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-17: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.112938] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-18: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.113041] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-19: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.113065] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-20: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.113096] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-21: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.113154] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-22: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.113207] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-23: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.113205] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 11:35:05.113258] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-24: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.113281] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 11:35:05.113334] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-25: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.113399] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-26: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.113405] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 11:35:05.113449] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-27: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.113507] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-28: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.113596] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-29: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.113637] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-30: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.113690] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-31: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.113743] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-32: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.113784] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-33: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.113803] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 11:35:05.113839] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-34: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.113909] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-35: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.114115] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 11:35:05.114141] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 11:35:05.134272] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-30: lk op is for a transaction >[2018-08-18 11:35:05.135267] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:05.135672] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-23: lk op is for a transaction >[2018-08-18 11:35:05.136103] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 11:35:05.137140] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 11:35:05.137424] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:05.137843] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-27: lk op is for a transaction >[2018-08-18 11:35:05.138011] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 11:35:05.138930] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 11:35:05.139095] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 11:35:05.139340] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 11:35:05.139397] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 11:35:05.139570] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 11:35:05.139677] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 11:35:05.139830] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 11:35:05.140715] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 11:35:05.140869] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 11:35:05.140948] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:05.141174] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 11:35:05.141362] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 11:35:05.141508] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-20: lk op is for a transaction >[2018-08-18 11:35:05.141545] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-26: lk op is for a transaction >[2018-08-18 11:35:05.142507] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 11:35:05.145087] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-22: lk op is for a transaction >[2018-08-18 11:35:05.145404] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 11:35:05.146101] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 11:35:05.148242] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:05.698462] D [MSGID: 0] [dht-selfheal.c:1879:dht_selfheal_layout_new_directory] 0-vol0-dht: chunk size = 0xffffffff / 274638528 = 15.638619 >[2018-08-18 11:35:05.698506] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-5 >[2018-08-18 11:35:05.698520] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-6 >[2018-08-18 11:35:05.698532] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-7 >[2018-08-18 11:35:05.698542] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-8 >[2018-08-18 11:35:05.698553] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-9 >[2018-08-18 11:35:05.698563] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-0 >[2018-08-18 11:35:05.698573] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-1 >[2018-08-18 11:35:05.698584] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-10 >[2018-08-18 11:35:05.698594] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-11 >[2018-08-18 11:35:05.698619] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-12 >[2018-08-18 11:35:05.698630] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-13 >[2018-08-18 11:35:05.698642] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-14 >[2018-08-18 11:35:05.698663] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-15 >[2018-08-18 11:35:05.698688] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-16 >[2018-08-18 11:35:05.698699] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-17 >[2018-08-18 11:35:05.698710] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-18 >[2018-08-18 11:35:05.698720] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-19 >[2018-08-18 11:35:05.698730] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-2 >[2018-08-18 11:35:05.698741] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-20 >[2018-08-18 11:35:05.698752] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-21 >[2018-08-18 11:35:05.698762] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-22 >[2018-08-18 11:35:05.698772] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-23 >[2018-08-18 11:35:05.698783] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-24 >[2018-08-18 11:35:05.698793] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-25 >[2018-08-18 11:35:05.698803] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-26 >[2018-08-18 11:35:05.698814] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-27 >[2018-08-18 11:35:05.698824] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-28 >[2018-08-18 11:35:05.698835] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-29 >[2018-08-18 11:35:05.698845] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-3 >[2018-08-18 11:35:05.698855] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-30 >[2018-08-18 11:35:05.698866] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-31 >[2018-08-18 11:35:05.698876] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-32 >[2018-08-18 11:35:05.698887] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-33 >[2018-08-18 11:35:05.698897] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-34 >[2018-08-18 11:35:05.698908] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-35 >[2018-08-18 11:35:05.698918] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-4 >[2018-08-18 11:35:05.699061] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 11:35:05.703908] D [MSGID: 109036] [dht-common.c:9587:dht_log_new_layout_for_dir_selfheal] 0-vol0-dht: Setting layout of /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m with [Subvol_name: vol0-replicate-0, Err: -1 , Start: 596523235 , Stop: 715827881 , Hash: 1 ], [Subvol_name: vol0-replicate-1, Err: -1 , Start: 715827882 , Stop: 835132528 , Hash: 1 ], [Subvol_name: vol0-replicate-10, Err: -1 , Start: 835132529 , Stop: 954437175 , Hash: 1 ], [Subvol_name: vol0-replicate-11, Err: -1 , Start: 954437176 , Stop: 1073741822 , Hash: 1 ], [Subvol_name: vol0-replicate-12, Err: -1 , Start: 1073741823 , Stop: 1193046469 , Hash: 1 ], [Subvol_name: vol0-replicate-13, Err: -1 , Start: 1193046470 , Stop: 1312351116 , Hash: 1 ], [Subvol_name: vol0-replicate-14, Err: -1 , Start: 1312351117 , Stop: 1431655763 , Hash: 1 ], [Subvol_name: vol0-replicate-15, Err: -1 , Start: 1431655764 , Stop: 1550960410 , Hash: 1 ], [Subvol_name: vol0-replicate-16, Err: -1 , Start: 1550960411 , Stop: 1670265057 , Hash: 1 ], [Subvol_name: vol0-replicate-17, Err: -1 , Start: 1670265058 , Stop: 1789569704 , Hash: 1 ], [Subvol_name: vol0-replicate-18, Err: -1 , Start: 1789569705 , Stop: 1908874351 , Hash: 1 ], [Subvol_name: vol0-replicate-19, Err: -1 , Start: 1908874352 , Stop: 2028178998 , Hash: 1 ], [Subvol_name: vol0-replicate-2, Err: -1 , Start: 2028178999 , Stop: 2147483645 , Hash: 1 ], [Subvol_name: vol0-replicate-20, Err: -1 , Start: 2147483646 , Stop: 2266788292 , Hash: 1 ], [Subvol_name: vol0-replicate-21, Err: -1 , Start: 2266788293 , Stop: 2386092939 , Hash: 1 ], [Subvol_name: vol0-replicate-22, Err: -1 , Start: 2386092940 , Stop: 2505397586 , Hash: 1 ], [Subvol_name: vol0-replicate-23, Err: -1 , Start: 2505397587 , Stop: 2624702233 , Hash: 1 ], [Subvol_name: vol0-replicate-24, Err: -1 , Start: 2624702234 , Stop: 2744006880 , Hash: 1 ], [Subvol_name: vol0-replicate-25, Err: -1 , Start: 2744006881 , Stop: 2863311527 , Hash: 1 ], [Subvol_name: vol0-replicate-26, Err: -1 , Start: 2863311528 , Stop: 2982616174 , Hash: 1 ], [Subvol_name: vol0-replicate-27, Err: -1 , Start: 2982616175 , Stop: 3101920821 , Hash: 1 ], [Subvol_name: vol0-replicate-28, Err: -1 , Start: 3101920822 , Stop: 3221225468 , Hash: 1 ], [Subvol_name: vol0-replicate-29, Err: -1 , Start: 3221225469 , Stop: 3340530115 , Hash: 1 ], [Subvol_name: vol0-replicate-3, Err: -1 , Start: 3340530116 , Stop: 3459834762 , Hash: 1 ], [Subvol_name: vol0-replicate-30, Err: -1 , Start: 3459834763 , Stop: 3579139409 , Hash: 1 ], [Subvol_name: vol0-replicate-31, Err: -1 , Start: 3579139410 , Stop: 3698444056 , Hash: 1 ], [Subvol_name: vol0-replicate-32, Err: -1 , Start: 3698444057 , Stop: 3817748703 , Hash: 1 ], [Subvol_name: vol0-replicate-33, Err: -1 , Start: 3817748704 , Stop: 3937053350 , Hash: 1 ], [Subvol_name: vol0-replicate-34, Err: -1 , Start: 3937053351 , Stop: 4056357997 , Hash: 1 ], [Subvol_name: vol0-replicate-35, Err: -1 , Start: 4056357998 , Stop: 4175662644 , Hash: 1 ], [Subvol_name: vol0-replicate-4, Err: -1 , Start: 4175662645 , Stop: 4294967295 , Hash: 1 ], [Subvol_name: vol0-replicate-5, Err: -1 , Start: 0 , Stop: 119304646 , Hash: 1 ], [Subvol_name: vol0-replicate-6, Err: -1 , Start: 119304647 , Stop: 238609293 , Hash: 1 ], [Subvol_name: vol0-replicate-7, Err: -1 , Start: 238609294 , Stop: 357913940 , Hash: 1 ], [Subvol_name: vol0-replicate-8, Err: -1 , Start: 357913941 , Stop: 477218587 , Hash: 1 ], [Subvol_name: vol0-replicate-9, Err: -1 , Start: 477218588 , Stop: 596523234 , Hash: 1 ], >[2018-08-18 11:35:05.704139] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-0: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.704156] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-2: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.704181] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd006dfc0, vol0-client-0 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.704139] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-1: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.704207] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd006dfc0, vol0-client-2 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.704255] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd006dfc0, vol0-client-1 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.704255] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-3: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.704277] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-5: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.704282] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-4: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.704290] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-30: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.704296] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 11:35:05.704333] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd0000d90, vol0-client-30 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.704293] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd009a150, vol0-client-3 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.704345] D [MSGID: 0] [afr-transaction.c:1877:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-0: Non blocking inodelks failed. Proceeding to blocking >[2018-08-18 11:35:05.704324] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd009a150, vol0-client-4 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.704367] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-32: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.704349] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd009a150, vol0-client-5 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.704381] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-31: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.704400] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd0000d90, vol0-client-31 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.704405] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 11:35:05.704398] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-33: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.704409] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd0000d90, vol0-client-32 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.704438] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd00e6f20, vol0-client-33 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.704440] D [MSGID: 0] [afr-transaction.c:1877:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-1: Non blocking inodelks failed. Proceeding to blocking >[2018-08-18 11:35:05.704452] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:05.704455] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-36: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.704480] D [MSGID: 0] [afr-transaction.c:1877:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks failed. Proceeding to blocking >[2018-08-18 11:35:05.704489] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd0013370, vol0-client-36 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.704501] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-35: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.704513] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-37: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.704536] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd0013370, vol0-client-37 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.704539] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-38: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.704504] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-34: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.704546] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd00e6f20, vol0-client-35 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.704568] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd00e6f20, vol0-client-34 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.704571] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd0013370, vol0-client-38 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.704573] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-41: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.704574] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-39: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.704595] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:05.704638] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd001bcf0, vol0-client-39 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.704629] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.704629] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd001bcf0, vol0-client-41 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.704611] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 11:35:05.704666] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 11:35:05.704663] D [MSGID: 0] [afr-transaction.c:1877:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks failed. Proceeding to blocking >[2018-08-18 11:35:05.704673] D [MSGID: 0] [afr-transaction.c:1877:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-12: Non blocking inodelks failed. Proceeding to blocking >[2018-08-18 11:35:05.704739] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-15: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.704771] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.704792] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-17: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.704818] D [MSGID: 0] [afr-transaction.c:1877:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks failed. Proceeding to blocking >[2018-08-18 11:35:05.704894] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-18: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.704959] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 11:35:05.704960] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-19: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.705023] D [MSGID: 0] [afr-lk-common.c:1189:afr_lock_blocking] 0-vol0-replicate-0: we're done locking >[2018-08-18 11:35:05.705028] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 11:35:05.705055] D [MSGID: 0] [afr-transaction.c:1857:afr_post_blocking_inodelk_cbk] 0-vol0-replicate-0: Blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.705079] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 11:35:05.705125] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-2: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.705176] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 11:35:05.705202] D [MSGID: 0] [afr-lk-common.c:1189:afr_lock_blocking] 0-vol0-replicate-1: we're done locking >[2018-08-18 11:35:05.705219] D [MSGID: 0] [afr-transaction.c:1857:afr_post_blocking_inodelk_cbk] 0-vol0-replicate-1: Blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.705267] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 11:35:05.705270] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-20: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.705349] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.705358] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 11:35:05.705400] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-22: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.705425] D [MSGID: 0] [afr-lk-common.c:1189:afr_lock_blocking] 0-vol0-replicate-10: we're done locking >[2018-08-18 11:35:05.705450] D [MSGID: 0] [afr-transaction.c:1857:afr_post_blocking_inodelk_cbk] 0-vol0-replicate-10: Blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.705476] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 11:35:05.705524] D [MSGID: 0] [afr-lk-common.c:1189:afr_lock_blocking] 0-vol0-replicate-11: we're done locking >[2018-08-18 11:35:05.705539] D [MSGID: 0] [afr-transaction.c:1857:afr_post_blocking_inodelk_cbk] 0-vol0-replicate-11: Blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.705542] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 11:35:05.705567] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-23: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.705582] D [MSGID: 0] [afr-lk-common.c:1189:afr_lock_blocking] 0-vol0-replicate-13: we're done locking >[2018-08-18 11:35:05.705633] D [MSGID: 0] [afr-transaction.c:1857:afr_post_blocking_inodelk_cbk] 0-vol0-replicate-13: Blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.705646] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 11:35:05.705696] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.705713] D [MSGID: 0] [afr-lk-common.c:1189:afr_lock_blocking] 0-vol0-replicate-12: we're done locking >[2018-08-18 11:35:05.705729] D [MSGID: 0] [afr-transaction.c:1857:afr_post_blocking_inodelk_cbk] 0-vol0-replicate-12: Blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.705792] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-25: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.705823] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-20: lk op is for a transaction >[2018-08-18 11:35:05.705849] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-26: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.705879] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 11:35:05.705924] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-27: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.705934] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-22: lk op is for a transaction >[2018-08-18 11:35:05.705973] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:05.706009] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-28: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.706080] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-23: lk op is for a transaction >[2018-08-18 11:35:05.706073] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:05.706114] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-29: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.706165] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 11:35:05.706181] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:05.706205] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-3: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.706279] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 11:35:05.706328] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-30: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.706349] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-26: lk op is for a transaction >[2018-08-18 11:35:05.706337] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 11:35:05.706454] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.706485] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-32: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.706490] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-27: lk op is for a transaction >[2018-08-18 11:35:05.706523] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 11:35:05.706571] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-33: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.706657] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 11:35:05.706658] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-34: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.706721] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 11:35:05.706821] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.706821] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-30: lk op is for a transaction >[2018-08-18 11:35:05.706865] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 11:35:05.706871] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-5: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.706965] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-6: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.706962] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 11:35:05.706985] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 11:35:05.706998] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.707083] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-8: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.707083] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 11:35:05.707081] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-35: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.707133] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-9: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.707184] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:05.707218] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 11:35:05.707272] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 11:35:05.707303] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 11:35:05.707351] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 11:35:05.707372] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 11:35:05.707586] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 11:35:05.708532] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88 on vol0-replicate-11 >[2018-08-18 11:35:05.708868] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.708863] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.708936] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.708950] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.708966] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88 with op_ret -1 [No such file or directory] >[2018-08-18 11:35:05.708975] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88 missing on subvol vol0-replicate-11 >[2018-08-18 11:35:05.708983] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:05.709244] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709288] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709288] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709334] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709339] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709370] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709366] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709391] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-1 >[2018-08-18 11:35:05.709382] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709420] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709419] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709458] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709463] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709466] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709428] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709468] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709500] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709515] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709525] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-2 >[2018-08-18 11:35:05.709527] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-0 >[2018-08-18 11:35:05.709532] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-3 >[2018-08-18 11:35:05.709557] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709575] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709581] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709604] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709627] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709606] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709634] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709652] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709658] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709670] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-5 >[2018-08-18 11:35:05.709680] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709690] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-6 >[2018-08-18 11:35:05.709709] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709714] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709749] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709752] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709746] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709757] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709777] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-7 >[2018-08-18 11:35:05.709783] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709786] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709797] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-4 >[2018-08-18 11:35:05.709809] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709826] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-8 >[2018-08-18 11:35:05.709840] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709841] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709867] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709870] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709890] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709912] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709924] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-9 >[2018-08-18 11:35:05.709938] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709946] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709962] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.709977] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-10 >[2018-08-18 11:35:05.710064] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710068] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710086] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710102] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710123] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710135] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-12 >[2018-08-18 11:35:05.710138] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710142] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710179] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710210] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710222] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710228] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-13 >[2018-08-18 11:35:05.710273] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710314] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710298] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710339] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710347] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710377] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-14 >[2018-08-18 11:35:05.710392] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710412] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710425] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-15 >[2018-08-18 11:35:05.710459] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710459] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710477] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710477] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710505] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710510] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710520] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-16 >[2018-08-18 11:35:05.710580] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710596] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710626] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-17 >[2018-08-18 11:35:05.710629] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710660] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710711] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710734] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710734] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710752] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-18 >[2018-08-18 11:35:05.710784] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710788] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710831] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710855] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710856] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-19 >[2018-08-18 11:35:05.710883] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710911] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710931] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710954] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-20 >[2018-08-18 11:35:05.710962] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.710968] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711068] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711108] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711126] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-21 >[2018-08-18 11:35:05.711131] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711158] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711217] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711234] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711246] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-22 >[2018-08-18 11:35:05.711248] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711249] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711251] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711287] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711304] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-23 >[2018-08-18 11:35:05.711375] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711410] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711437] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711443] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711467] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711482] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711484] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-24 >[2018-08-18 11:35:05.711544] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711564] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711564] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711611] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711619] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711639] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-26 >[2018-08-18 11:35:05.711658] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711676] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-25 >[2018-08-18 11:35:05.711679] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711699] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711699] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711736] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711743] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711770] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-27 >[2018-08-18 11:35:05.711784] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711797] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711824] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711839] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-28 >[2018-08-18 11:35:05.711842] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711887] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711898] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711914] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711924] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-29 >[2018-08-18 11:35:05.711945] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711948] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.711994] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712027] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712037] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712043] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-30 >[2018-08-18 11:35:05.712043] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712084] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712095] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712124] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712127] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712142] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712145] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-31 >[2018-08-18 11:35:05.712171] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712184] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-32 >[2018-08-18 11:35:05.712230] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712258] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712262] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712291] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712306] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-33 >[2018-08-18 11:35:05.712296] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712346] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712357] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712374] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-11 >[2018-08-18 11:35:05.712405] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712405] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712425] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712437] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712445] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712450] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712462] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-34 >[2018-08-18 11:35:05.712471] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712488] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88) from subvol vol0-replicate-35 >[2018-08-18 11:35:05.712504] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-11 cached_subvol null >[2018-08-18 11:35:05.712517] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88 >[2018-08-18 11:35:05.712529] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712544] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712560] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712575] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712589] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfb00034a0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.712766] D [MSGID: 0] [dht-common.c:6923:dht_create_wind_to_avail_subvol] 0-vol0-dht: creating /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88 on vol0-replicate-11 >[2018-08-18 11:35:05.713046] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-11: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.726139] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-vol0-dht: Found anomalies in /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m (gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971). Holes=2 overlaps=0 >[2018-08-18 11:35:05.726171] D [MSGID: 0] [dht-common.c:943:dht_lookup_dir_cbk] 0-vol0-dht: fixing assignment on /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m >[2018-08-18 11:35:05.726229] D [MSGID: 0] [dht-selfheal.c:1879:dht_selfheal_layout_new_directory] 0-vol0-dht: chunk size = 0xffffffff / 274638528 = 15.638619 >[2018-08-18 11:35:05.726248] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-34 >[2018-08-18 11:35:05.726260] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-35 >[2018-08-18 11:35:05.726271] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-4 >[2018-08-18 11:35:05.726283] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-5 >[2018-08-18 11:35:05.726293] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-6 >[2018-08-18 11:35:05.726304] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-7 >[2018-08-18 11:35:05.726314] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-8 >[2018-08-18 11:35:05.726325] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-9 >[2018-08-18 11:35:05.726335] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-0 >[2018-08-18 11:35:05.726346] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-1 >[2018-08-18 11:35:05.726356] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-10 >[2018-08-18 11:35:05.726367] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-11 >[2018-08-18 11:35:05.726378] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-12 >[2018-08-18 11:35:05.726389] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-13 >[2018-08-18 11:35:05.726399] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-14 >[2018-08-18 11:35:05.726410] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-15 >[2018-08-18 11:35:05.726421] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-16 >[2018-08-18 11:35:05.726431] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-17 >[2018-08-18 11:35:05.726442] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-18 >[2018-08-18 11:35:05.726453] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-19 >[2018-08-18 11:35:05.726470] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-2 >[2018-08-18 11:35:05.726481] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-20 >[2018-08-18 11:35:05.726492] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-21 >[2018-08-18 11:35:05.726502] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-22 >[2018-08-18 11:35:05.726513] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-23 >[2018-08-18 11:35:05.726524] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-24 >[2018-08-18 11:35:05.726534] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-25 >[2018-08-18 11:35:05.726545] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-26 >[2018-08-18 11:35:05.726594] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-27 >[2018-08-18 11:35:05.726618] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-28 >[2018-08-18 11:35:05.726629] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-29 >[2018-08-18 11:35:05.726660] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-3 >[2018-08-18 11:35:05.726676] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-30 >[2018-08-18 11:35:05.726687] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-31 >[2018-08-18 11:35:05.726698] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-32 >[2018-08-18 11:35:05.726709] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-33 >[2018-08-18 11:35:05.726949] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-0: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.726969] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-1: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.727005] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbf98027e10, vol0-client-1 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.727015] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbf98027e10, vol0-client-0 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.727010] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-2: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.727056] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbf98027e10, vol0-client-2 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.731905] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-vol0-dht: Found anomalies in /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m (gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971). Holes=2 overlaps=0 >[2018-08-18 11:35:05.731940] D [MSGID: 0] [dht-common.c:943:dht_lookup_dir_cbk] 0-vol0-dht: fixing assignment on /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m >[2018-08-18 11:35:05.731996] D [MSGID: 0] [dht-selfheal.c:1879:dht_selfheal_layout_new_directory] 0-vol0-dht: chunk size = 0xffffffff / 274638528 = 15.638619 >[2018-08-18 11:35:05.732013] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-34 >[2018-08-18 11:35:05.732024] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-35 >[2018-08-18 11:35:05.732035] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-4 >[2018-08-18 11:35:05.732046] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-5 >[2018-08-18 11:35:05.732055] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-6 >[2018-08-18 11:35:05.732065] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-7 >[2018-08-18 11:35:05.732074] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-8 >[2018-08-18 11:35:05.732084] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-9 >[2018-08-18 11:35:05.732094] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-0 >[2018-08-18 11:35:05.732103] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-1 >[2018-08-18 11:35:05.732112] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-10 >[2018-08-18 11:35:05.732122] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-11 >[2018-08-18 11:35:05.732131] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-12 >[2018-08-18 11:35:05.732141] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-13 >[2018-08-18 11:35:05.732150] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-14 >[2018-08-18 11:35:05.732160] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-15 >[2018-08-18 11:35:05.732170] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-16 >[2018-08-18 11:35:05.732179] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-17 >[2018-08-18 11:35:05.732189] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-18 >[2018-08-18 11:35:05.732198] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-19 >[2018-08-18 11:35:05.732207] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-2 >[2018-08-18 11:35:05.732217] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-20 >[2018-08-18 11:35:05.732227] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-21 >[2018-08-18 11:35:05.732240] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-22 >[2018-08-18 11:35:05.732250] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-23 >[2018-08-18 11:35:05.732259] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-24 >[2018-08-18 11:35:05.732269] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-25 >[2018-08-18 11:35:05.732279] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-26 >[2018-08-18 11:35:05.732288] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-27 >[2018-08-18 11:35:05.732298] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-28 >[2018-08-18 11:35:05.732308] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-29 >[2018-08-18 11:35:05.732317] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-3 >[2018-08-18 11:35:05.732327] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-30 >[2018-08-18 11:35:05.732336] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-31 >[2018-08-18 11:35:05.732346] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-32 >[2018-08-18 11:35:05.732355] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-33 >[2018-08-18 11:35:05.732655] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-0: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.732667] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-2: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.732655] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-1: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.732723] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbf84043830, vol0-client-0 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.732734] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbf84043830, vol0-client-1 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.732723] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbf84043830, vol0-client-2 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.735545] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-11 >[2018-08-18 11:35:05.735645] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:05.770671] D [MSGID: 0] [dht-selfheal.c:1206:dht_selfheal_dir_mkdir_setacl] 0-vol0-dht: ACL_DEFAULT xattr not present >[2018-08-18 11:35:05.770695] D [MSGID: 0] [dht-selfheal.c:1219:dht_selfheal_dir_mkdir_setacl] 0-vol0-dht: ACL_ACCESS xattr not present >[2018-08-18 11:35:05.770706] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-0 >[2018-08-18 11:35:05.770792] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-10 >[2018-08-18 11:35:05.770868] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-11 >[2018-08-18 11:35:05.770894] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-0: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.770932] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-12 >[2018-08-18 11:35:05.770958] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-10: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.770998] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-13 >[2018-08-18 11:35:05.771032] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-0: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771033] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-11: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.771063] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc000ac0, vol0-client-0 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771068] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-1: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771083] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-14 >[2018-08-18 11:35:05.771078] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-2: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771079] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-30: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771102] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc000ac0, vol0-client-1 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771131] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc6c8b0, vol0-client-30 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771120] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc000ac0, vol0-client-2 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771143] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-31: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771161] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-15 >[2018-08-18 11:35:05.771171] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-32: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771167] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-12: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.771164] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc6c8b0, vol0-client-31 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771187] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 11:35:05.771198] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc6c8b0, vol0-client-32 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771207] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-33: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771238] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-17 >[2018-08-18 11:35:05.771240] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-13: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.771240] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc32de0, vol0-client-33 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771238] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:05.771266] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-34: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771271] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-35: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771288] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc32de0, vol0-client-34 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771293] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc32de0, vol0-client-35 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771307] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-18 >[2018-08-18 11:35:05.771311] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:05.771324] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-36: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771325] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-14: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.771348] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc387a0, vol0-client-36 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771357] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-38: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771358] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-37: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771378] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc387a0, vol0-client-38 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771399] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-39: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771406] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc387a0, vol0-client-37 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771403] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-19 >[2018-08-18 11:35:05.771428] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 11:35:05.771431] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc009cb0, vol0-client-39 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771426] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-15: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.771449] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-0 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771454] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-10 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771460] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-40: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771474] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.771483] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc009cb0, vol0-client-40 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771486] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.771506] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-41: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771507] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-2 >[2018-08-18 11:35:05.771521] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc009cb0, vol0-client-41 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771529] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-42: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771538] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 11:35:05.771535] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-43: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771536] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-17: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.771565] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc0039f0, vol0-client-43 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771542] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc0039f0, vol0-client-42 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771578] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-11 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771603] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-18: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.771607] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.771610] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-21 >[2018-08-18 11:35:05.771634] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-45: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771646] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-44: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771650] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-46: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771662] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc0039f0, vol0-client-44 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771650] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc00f590, vol0-client-45 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771678] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc00f590, vol0-client-46 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771680] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 11:35:05.771699] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-19: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.771703] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-23 >[2018-08-18 11:35:05.771700] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-47: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771699] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-12 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771729] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-51: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771730] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc00f590, vol0-client-47 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771755] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc2c840, vol0-client-51 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771769] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 11:35:05.771774] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-2: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.771779] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.771785] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-52: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771770] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-28 >[2018-08-18 11:35:05.771811] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-13 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771816] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc2c840, vol0-client-52 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771835] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.771839] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-53: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771849] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-54: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771859] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc2c840, vol0-client-53 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771862] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-56: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771859] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-55: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771884] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc11020, vol0-client-56 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771863] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc11020, vol0-client-54 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771886] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 11:35:05.771891] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-21: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.771876] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-29 >[2018-08-18 11:35:05.771893] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc11020, vol0-client-55 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771937] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-58: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771949] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 11:35:05.771955] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-59: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771952] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc20da0, vol0-client-58 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771969] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc20da0, vol0-client-59 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771986] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-57: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771996] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-6: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.771998] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-14 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.771992] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-23: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.771990] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-3 >[2018-08-18 11:35:05.772025] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-7: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772025] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.772032] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc20da0, vol0-client-57 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772057] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc6c8b0, vol0-client-7 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772059] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 11:35:05.772064] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc6c8b0, vol0-client-6 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772076] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-8: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772096] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc6c8b0, vol0-client-8 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772101] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-15 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772096] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-28: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.772112] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-64: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772126] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc32de0, vol0-client-64 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772128] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 11:35:05.772110] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-30 >[2018-08-18 11:35:05.772133] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.772148] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-65: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772160] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc32de0, vol0-client-65 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772158] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-63: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772182] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc32de0, vol0-client-63 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772189] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-17 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772198] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 11:35:05.772210] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-32 >[2018-08-18 11:35:05.772228] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.772222] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-29: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.772234] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-18 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772237] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-70: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772269] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc000ac0, vol0-client-70 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772271] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.772271] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-71: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772295] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-33 >[2018-08-18 11:35:05.772302] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc000ac0, vol0-client-71 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772295] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-69: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772322] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-19 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772325] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc000ac0, vol0-client-69 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772330] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-86: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772325] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-84: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772338] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.772326] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-3: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.772355] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-23: lk op is for a transaction >[2018-08-18 11:35:05.772355] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc387a0, vol0-client-84 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772358] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc387a0, vol0-client-86 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772368] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-34 >[2018-08-18 11:35:05.772381] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-85: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772403] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc387a0, vol0-client-85 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772416] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 11:35:05.772421] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-30: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.772416] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-2 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772452] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-87: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772456] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.772457] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-35 >[2018-08-18 11:35:05.772463] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-89: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772483] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc009cb0, vol0-client-87 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772487] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc009cb0, vol0-client-89 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772498] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-21 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772502] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-32: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.772504] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-88: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772522] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-9: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772527] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc009cb0, vol0-client-88 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772524] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.772527] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-33: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.772546] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-4 >[2018-08-18 11:35:05.772557] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc6afb0, vol0-client-9 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772547] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 11:35:05.772574] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-10: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772583] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-11: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772588] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc6afb0, vol0-client-10 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772613] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc6afb0, vol0-client-11 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772630] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-5 >[2018-08-18 11:35:05.772644] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 11:35:05.772629] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-23 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772638] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-90: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772673] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc0039f0, vol0-client-90 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772681] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-91: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772690] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.772693] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-34: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.772699] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc0039f0, vol0-client-91 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772702] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-9 >[2018-08-18 11:35:05.772712] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-28 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772730] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-96: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772730] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-92: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772743] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.772757] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc0039f0, vol0-client-92 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772757] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-97: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772759] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc00f590, vol0-client-96 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772793] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc00f590, vol0-client-97 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772794] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-98: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772814] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-100: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772823] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc00f590, vol0-client-98 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772824] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-30: lk op is for a transaction >[2018-08-18 11:35:05.772815] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-99: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772830] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc11020, vol0-client-100 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772845] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc11020, vol0-client-99 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772831] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-101: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772862] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 11:35:05.772864] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-29 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772874] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc11020, vol0-client-101 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772881] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.772874] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-4: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.772892] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 11:35:05.772904] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-5: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.772913] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-102: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772921] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-3 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772930] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc20da0, vol0-client-102 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772934] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-35: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.772937] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.772954] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-103: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772967] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc20da0, vol0-client-103 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.772983] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-104: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.772989] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-9: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.773003] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc20da0, vol0-client-104 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.773008] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-12: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.773005] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-30 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.773027] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-15: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.773029] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc32de0, vol0-client-12 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.773049] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc2c840, vol0-client-15 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.773040] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-13: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.773042] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.773066] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-17: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.773030] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 11:35:05.773070] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-16: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.773086] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc2c840, vol0-client-17 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.773073] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc32de0, vol0-client-13 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.773090] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-32 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.773120] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.773122] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-105: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.773129] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-33 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.773145] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc6c8b0, vol0-client-105 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.773121] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc2c840, vol0-client-16 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.773126] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-14: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.773169] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-28: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.773171] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-106: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.773184] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc000ac0, vol0-client-28 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.773186] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-27: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.773170] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 11:35:05.773184] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.773185] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc32de0, vol0-client-14 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.773204] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc000ac0, vol0-client-27 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.773193] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc6c8b0, vol0-client-106 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.773219] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-29: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.773231] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-107: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.773234] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdc000ac0, vol0-client-29 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.773235] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:05.773263] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 11:35:05.773247] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfdcc6c8b0, vol0-client-107 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.773263] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-34 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.773298] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 11:35:05.773301] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.773371] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-5 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.773405] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.773422] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-4 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.773441] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.773429] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-9 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.773472] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbfa800f430, vol0-replicate-35 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.773477] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.777618] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 11:35:05.773491] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.777616] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-1: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.777722] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-0: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.777868] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 11:35:05.777921] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 11:35:05.778041] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.778174] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.778240] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:05.778332] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-12: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.778389] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:05.778511] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.778575] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 11:35:05.778626] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.778734] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 11:35:05.778746] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.778805] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 11:35:05.778826] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-15: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.778956] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 11:35:05.779018] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 11:35:05.779185] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-17: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.779325] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-18: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.779388] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 11:35:05.779510] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-19: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.779535] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 11:35:05.779555] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-20: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.779617] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-2: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.779705] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 11:35:05.779739] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-20: lk op is for a transaction >[2018-08-18 11:35:05.779823] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 11:35:05.779846] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-22: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.780019] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.780047] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-22: lk op is for a transaction >[2018-08-18 11:35:05.780159] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.780199] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 11:35:05.780279] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-25: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.780284] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-23: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.780348] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:05.780420] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-26: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.780476] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 11:35:05.780496] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-23: lk op is for a transaction >[2018-08-18 11:35:05.780536] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-27: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.780636] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-26: lk op is for a transaction >[2018-08-18 11:35:05.780707] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-27: lk op is for a transaction >[2018-08-18 11:35:05.780935] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-28: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.781103] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 11:35:05.781141] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-29: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.781242] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-3: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.781298] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.781366] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 11:35:05.781441] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 11:35:05.781459] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-30: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.781494] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 11:35:05.781651] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-30: lk op is for a transaction >[2018-08-18 11:35:05.781706] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-32: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.781898] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 11:35:05.781901] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-33: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.782062] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-34: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.782089] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 11:35:05.782253] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-35: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.782254] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 11:35:05.782398] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.782452] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-6: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.782449] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 11:35:05.782587] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:05.782627] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.782644] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-5: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.782663] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 11:35:05.782752] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-8: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.782831] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 11:35:05.782834] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 11:35:05.782916] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 11:35:05.783194] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-9: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.783462] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 11:35:05.783543] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-0: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.783565] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd805b530, vol0-client-0 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.783569] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-2: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.783566] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-1: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.783604] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd805b530, vol0-client-2 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.783611] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd805b530, vol0-client-1 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.820319] D [MSGID: 0] [dht-selfheal.c:1206:dht_selfheal_dir_mkdir_setacl] 0-vol0-dht: ACL_DEFAULT xattr not present >[2018-08-18 11:35:05.820351] D [MSGID: 0] [dht-selfheal.c:1219:dht_selfheal_dir_mkdir_setacl] 0-vol0-dht: ACL_ACCESS xattr not present >[2018-08-18 11:35:05.820365] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-0 >[2018-08-18 11:35:05.820475] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-10 >[2018-08-18 11:35:05.820562] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-14 >[2018-08-18 11:35:05.820575] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-0: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.820680] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-15 >[2018-08-18 11:35:05.820681] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-10: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.820739] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-0: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.820770] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8049560, vol0-client-0 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.820770] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-14: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.820775] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-1: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.820795] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-2: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.820820] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8049560, vol0-client-1 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.820823] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-17 >[2018-08-18 11:35:05.820841] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8049560, vol0-client-2 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.820871] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-31: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.820854] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-30: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.820893] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd80135a0, vol0-client-31 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.820906] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd80135a0, vol0-client-30 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.820909] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-32: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.820913] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 11:35:05.820913] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-15: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.820959] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd80135a0, vol0-client-32 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.820980] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-43: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.820981] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:05.820991] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-42: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.820991] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-18 >[2018-08-18 11:35:05.821003] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8045cf0, vol0-client-43 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821026] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8045cf0, vol0-client-42 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821082] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-17: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.821075] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-44: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.821095] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-46: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.821104] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-47: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.821106] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8045cf0, vol0-client-44 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821112] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8044cd0, vol0-client-46 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821125] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8044cd0, vol0-client-47 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821097] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-45: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.821137] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 11:35:05.821170] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-0 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821173] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-10 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821172] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8044cd0, vol0-client-45 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821192] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-51: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.821166] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-19 >[2018-08-18 11:35:05.821224] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.821232] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.821233] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-52: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.821228] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8042240, vol0-client-51 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821219] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 11:35:05.821262] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8042240, vol0-client-52 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821275] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-53: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.821282] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-18: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.821289] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8042240, vol0-client-53 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821305] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-14 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821306] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 11:35:05.821328] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.821339] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-2 >[2018-08-18 11:35:05.821398] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-54: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.821422] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8048540, vol0-client-54 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821429] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-19: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.821429] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-55: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.821444] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-56: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.821460] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8048540, vol0-client-55 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821463] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8048540, vol0-client-56 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821464] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-17 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821474] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-15 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821482] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-29 >[2018-08-18 11:35:05.821535] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.821540] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-57: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.821520] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 11:35:05.821536] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.821563] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd804ad90, vol0-client-57 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821569] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-58: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.821591] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-59: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.821588] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd804ad90, vol0-client-58 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821619] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd804ad90, vol0-client-59 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821593] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-2: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.821639] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 11:35:05.821645] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-3 >[2018-08-18 11:35:05.821710] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-18 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821729] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.821747] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-7: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.821738] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-29: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.821755] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-6: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.821764] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8045cf0, vol0-client-7 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821767] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-33 >[2018-08-18 11:35:05.821791] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-8: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.821802] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8045cf0, vol0-client-6 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821827] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8045cf0, vol0-client-8 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821808] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-19 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821843] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 11:35:05.821869] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-3: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.821872] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.821890] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-4 >[2018-08-18 11:35:05.821900] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-87: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.821914] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8049560, vol0-client-87 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821932] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-88: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.821952] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8049560, vol0-client-88 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.821980] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-9: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.821988] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m on subvol vol0-replicate-9 >[2018-08-18 11:35:05.822013] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-10: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.822009] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8044cd0, vol0-client-9 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.822013] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-33: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.822077] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8044cd0, vol0-client-10 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.822078] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-89: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.822083] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-11: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.822090] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-2 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.822113] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8044cd0, vol0-client-11 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.822113] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8049560, vol0-client-89 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.822139] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-4: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.822143] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 11:35:05.822132] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.822146] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 11:35:05.822183] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-99: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.822196] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8048540, vol0-client-99 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.822203] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-101: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.822211] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-100: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.822223] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8048540, vol0-client-101 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.822234] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8048540, vol0-client-100 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.822241] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-9: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.822257] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 11:35:05.822261] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-12: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.822280] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd804ad90, vol0-client-12 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.822287] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-13: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.822322] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-14: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.822328] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd804ad90, vol0-client-13 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.822343] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd804ad90, vol0-client-14 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.822359] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:05.822371] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-27: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.822361] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-29 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.822373] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-28: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.822366] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-3 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.822375] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-29: remote operation failed. Path: /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m [File exists] >[2018-08-18 11:35:05.822387] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8042240, vol0-client-27 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.822430] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8042240, vol0-client-29 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.822439] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-33 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.822435] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7fbfd8042240, vol0-client-28 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.822442] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.822486] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 11:35:05.822469] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.822491] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.822485] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 11:35:05.822536] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-4 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.822555] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.822696] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7fbf9c033450, vol0-replicate-9 returned -1 error: File exists [File exists] >[2018-08-18 11:35:05.822716] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m, gfid = cf50d6fa-6d65-41aa-912b-17b59b78d971 [File exists] >[2018-08-18 11:35:05.824443] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-1: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.824552] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.824590] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-12: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.824623] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-0: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.824667] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 11:35:05.824664] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.824760] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:05.824838] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.824848] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 11:35:05.824876] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.824897] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 11:35:05.824984] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 11:35:05.825008] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.825142] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-15: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.825185] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 11:35:05.825192] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 11:35:05.825359] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-20: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.825363] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 11:35:05.825402] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.825458] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 11:35:05.825451] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-17: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.825514] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-22: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.825559] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-18: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.825580] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-23: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.825629] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-19: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.825672] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.825691] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-25: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.825740] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-20: lk op is for a transaction >[2018-08-18 11:35:05.825752] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-2: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.825792] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-26: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.825832] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-27: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.825852] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 11:35:05.825894] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-28: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.825955] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-22: lk op is for a transaction >[2018-08-18 11:35:05.825957] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 11:35:05.825989] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 11:35:05.826012] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-23: lk op is for a transaction >[2018-08-18 11:35:05.826099] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:05.826107] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 11:35:05.826127] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 11:35:05.826208] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-30: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.826257] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.826256] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 11:35:05.826293] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-32: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.826338] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-27: lk op is for a transaction >[2018-08-18 11:35:05.826350] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-26: lk op is for a transaction >[2018-08-18 11:35:05.826378] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 11:35:05.826436] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-34: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.826503] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-35: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.826545] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-29: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.826585] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-3: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.826587] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-5: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.826644] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-6: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.826690] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-30: lk op is for a transaction >[2018-08-18 11:35:05.826708] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 11:35:05.826731] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.826780] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-8: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.826783] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 11:35:05.826853] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-33: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.826878] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 11:35:05.826882] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 11:35:05.826911] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.826918] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 11:35:05.826915] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 11:35:05.826974] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 11:35:05.827016] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 11:35:05.827047] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 11:35:05.827063] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 11:35:05.827106] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 11:35:05.827155] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:05.827189] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-9: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:05.827452] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 11:35:05.827550] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-0: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.827556] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-2: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.827563] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-1: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.827577] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd408f2e0, vol0-client-0 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.827618] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd408f2e0, vol0-client-1 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.827619] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd408f2e0, vol0-client-2 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.877144] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-44: remote operation failed [Resource temporarily unavailable] >[2018-08-18 11:35:05.877174] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7fbfd805b530, vol0-client-44 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 11:35:05.890141] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88 on vol0-replicate-13 >[2018-08-18 11:35:05.890427] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.890472] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.890479] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.890510] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.890524] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88 with op_ret -1 [No such file or directory] >[2018-08-18 11:35:05.890537] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88 missing on subvol vol0-replicate-13 >[2018-08-18 11:35:05.890548] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:05.890783] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.890823] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.890828] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.890869] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.890877] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.890881] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.890932] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-0 >[2018-08-18 11:35:05.890934] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.890972] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.890971] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.890972] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.890990] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-1 >[2018-08-18 11:35:05.890978] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891024] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891030] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891038] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891054] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-2 >[2018-08-18 11:35:05.891066] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891086] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891095] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-3 >[2018-08-18 11:35:05.891135] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891148] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891170] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891211] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891234] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891235] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-4 >[2018-08-18 11:35:05.891239] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891256] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891274] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891282] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-5 >[2018-08-18 11:35:05.891339] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891349] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891370] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891391] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891401] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-6 >[2018-08-18 11:35:05.891433] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891441] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891462] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891471] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891477] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-7 >[2018-08-18 11:35:05.891515] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891531] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891548] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891572] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891580] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891586] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-8 >[2018-08-18 11:35:05.891605] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891666] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891663] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891688] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891703] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891724] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891724] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-9 >[2018-08-18 11:35:05.891752] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891758] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891759] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891783] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-10 >[2018-08-18 11:35:05.891814] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891836] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891837] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891854] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-11 >[2018-08-18 11:35:05.891874] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891916] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891926] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891934] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-12 >[2018-08-18 11:35:05.891959] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.891979] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892052] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892053] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892085] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892100] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892109] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-14 >[2018-08-18 11:35:05.892136] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892160] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892160] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892196] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892206] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-15 >[2018-08-18 11:35:05.892211] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892220] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892227] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892240] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892248] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-16 >[2018-08-18 11:35:05.892261] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892275] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892304] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892325] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892333] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892349] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-17 >[2018-08-18 11:35:05.892382] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892376] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892412] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892417] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892429] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-18 >[2018-08-18 11:35:05.892473] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892493] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892515] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892523] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892518] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892534] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892548] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-19 >[2018-08-18 11:35:05.892548] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892565] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-20 >[2018-08-18 11:35:05.892578] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892608] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892618] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892639] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892646] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-21 >[2018-08-18 11:35:05.892667] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892692] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892721] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892758] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892766] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892763] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892795] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-22 >[2018-08-18 11:35:05.892838] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892842] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892850] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892858] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-23 >[2018-08-18 11:35:05.892874] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892887] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892908] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892921] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-24 >[2018-08-18 11:35:05.892950] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.892977] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893005] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893019] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893028] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-25 >[2018-08-18 11:35:05.893028] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893047] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893093] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893113] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893119] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893126] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-26 >[2018-08-18 11:35:05.893143] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893158] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893173] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893181] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-27 >[2018-08-18 11:35:05.893220] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893221] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893276] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893295] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893298] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893318] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893333] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-29 >[2018-08-18 11:35:05.893318] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893363] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893364] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893372] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893397] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893400] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-28 >[2018-08-18 11:35:05.893408] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893410] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893427] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-30 >[2018-08-18 11:35:05.893438] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893488] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893505] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893516] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893517] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893524] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-31 >[2018-08-18 11:35:05.893596] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893589] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893615] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893643] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893650] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893652] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893663] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893654] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-32 >[2018-08-18 11:35:05.893681] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893683] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893682] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893705] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-33 >[2018-08-18 11:35:05.893713] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-13 >[2018-08-18 11:35:05.893729] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893722] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893740] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893747] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893764] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-34 >[2018-08-18 11:35:05.893800] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893825] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893834] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88) from subvol vol0-replicate-35 >[2018-08-18 11:35:05.893845] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-13 cached_subvol null >[2018-08-18 11:35:05.893852] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88 >[2018-08-18 11:35:05.893860] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893868] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893878] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893887] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.893896] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa000a2b0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.894203] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-0': avail_percent is: 96.00 and avail_space is: 7682550996992 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.894233] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-1': avail_percent is: 95.00 and avail_space is: 7649217478656 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.894322] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-2': avail_percent is: 96.00 and avail_space is: 7679628898304 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.894337] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-3': avail_percent is: 95.00 and avail_space is: 7676714049536 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.894377] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-4': avail_percent is: 96.00 and avail_space is: 7679852838912 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.894461] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-5': avail_percent is: 95.00 and avail_space is: 7672041328640 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.894482] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-6': avail_percent is: 94.00 and avail_space is: 7567700049920 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.894531] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-7': avail_percent is: 95.00 and avail_space is: 7599561781248 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.894635] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-8': avail_percent is: 95.00 and avail_space is: 7669932085248 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.894683] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-9': avail_percent is: 95.00 and avail_space is: 7675205881856 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.894747] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-10': avail_percent is: 94.00 and avail_space is: 7587254632448 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.894796] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-11': avail_percent is: 95.00 and avail_space is: 7666188136448 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.894850] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-12': avail_percent is: 95.00 and avail_space is: 7662336352256 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.894902] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-13': avail_percent is: 95.00 and avail_space is: 7671819669504 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.894956] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-14': avail_percent is: 95.00 and avail_space is: 7665784659968 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.895012] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-15': avail_percent is: 95.00 and avail_space is: 7654391312384 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.895044] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-16': avail_percent is: 95.00 and avail_space is: 7678806634496 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.895103] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-17': avail_percent is: 95.00 and avail_space is: 7669472206848 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.895176] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-18': avail_percent is: 95.00 and avail_space is: 7615164923904 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.895205] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-19': avail_percent is: 95.00 and avail_space is: 7660373045248 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.895241] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-20': avail_percent is: 95.00 and avail_space is: 7610263240704 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.895286] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-21': avail_percent is: 95.00 and avail_space is: 7663397208064 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.895332] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-22': avail_percent is: 95.00 and avail_space is: 7668074450944 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.895420] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-24': avail_percent is: 95.00 and avail_space is: 7660807667712 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.895464] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-25': avail_percent is: 95.00 and avail_space is: 7644131004416 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.895484] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-23': avail_percent is: 95.00 and avail_space is: 7669677629440 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.895511] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-26': avail_percent is: 95.00 and avail_space is: 7655957213184 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.895562] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-27': avail_percent is: 95.00 and avail_space is: 7652157960192 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.895632] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-28': avail_percent is: 95.00 and avail_space is: 7652197793792 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.895666] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-29': avail_percent is: 95.00 and avail_space is: 7639356854272 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.895784] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-31': avail_percent is: 95.00 and avail_space is: 7678164856832 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.895801] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-30': avail_percent is: 95.00 and avail_space is: 7675637026816 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.895818] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-32': avail_percent is: 95.00 and avail_space is: 7659078995968 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.895856] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-33': avail_percent is: 95.00 and avail_space is: 7676538961920 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.895874] D [MSGID: 0] [dht-common.c:6923:dht_create_wind_to_avail_subvol] 0-vol0-dht: creating /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88 on vol0-replicate-13 >[2018-08-18 11:35:05.895945] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-34': avail_percent is: 95.00 and avail_space is: 7665343188992 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.895967] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-35': avail_percent is: 95.00 and avail_space is: 7661850447872 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.896071] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-13: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.921002] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-13 >[2018-08-18 11:35:05.921049] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 11:35:05.924638] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88 on vol0-replicate-19 >[2018-08-18 11:35:05.924916] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.924946] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.924947] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.924993] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925006] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88 with op_ret -1 [No such file or directory] >[2018-08-18 11:35:05.925018] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88 missing on subvol vol0-replicate-19 >[2018-08-18 11:35:05.925028] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:05.925292] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925302] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925285] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925347] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925348] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925369] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925369] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925392] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-0 >[2018-08-18 11:35:05.925403] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925409] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925403] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925433] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-1 >[2018-08-18 11:35:05.925448] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925478] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925486] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925493] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925499] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-2 >[2018-08-18 11:35:05.925523] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925542] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925568] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-3 >[2018-08-18 11:35:05.925612] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925635] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925659] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925695] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925706] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925722] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925724] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-4 >[2018-08-18 11:35:05.925736] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925795] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925798] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925793] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925822] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-5 >[2018-08-18 11:35:05.925833] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925854] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925858] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925869] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-6 >[2018-08-18 11:35:05.925863] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925913] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925931] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925933] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925937] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925966] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-7 >[2018-08-18 11:35:05.925967] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925989] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.925999] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-8 >[2018-08-18 11:35:05.926024] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926030] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926066] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926076] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926095] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-9 >[2018-08-18 11:35:05.926098] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926120] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926125] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926146] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926162] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-10 >[2018-08-18 11:35:05.926207] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926207] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926223] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926246] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926260] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-11 >[2018-08-18 11:35:05.926273] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926294] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926301] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926326] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926344] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-12 >[2018-08-18 11:35:05.926354] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926384] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926401] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926436] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926442] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926445] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926456] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926493] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-13 >[2018-08-18 11:35:05.926516] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926516] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926535] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926555] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-14 >[2018-08-18 11:35:05.926518] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926595] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926607] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926609] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926612] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926635] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926666] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926663] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926664] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-15 >[2018-08-18 11:35:05.926665] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926706] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-16 >[2018-08-18 11:35:05.926720] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926746] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-17 >[2018-08-18 11:35:05.926719] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926756] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926772] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926798] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926803] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926845] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926840] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-18 >[2018-08-18 11:35:05.926842] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926871] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926890] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926912] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926926] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-20 >[2018-08-18 11:35:05.926936] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926969] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.926998] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927020] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927044] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-21 >[2018-08-18 11:35:05.927046] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927053] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927077] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927115] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927123] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927119] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927155] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-22 >[2018-08-18 11:35:05.927161] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927186] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927191] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927198] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927218] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-23 >[2018-08-18 11:35:05.927229] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927238] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927258] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927262] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927264] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927287] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-24 >[2018-08-18 11:35:05.927295] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927305] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-25 >[2018-08-18 11:35:05.927311] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927319] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927370] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927374] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927389] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927405] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927411] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927421] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927442] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-26 >[2018-08-18 11:35:05.927444] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-27 >[2018-08-18 11:35:05.927449] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927472] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927493] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927514] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927522] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-28 >[2018-08-18 11:35:05.927577] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927583] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927594] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927621] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927630] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-29 >[2018-08-18 11:35:05.927662] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927675] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927688] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927701] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927709] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-30 >[2018-08-18 11:35:05.927734] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927753] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927804] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927825] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927828] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927852] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-31 >[2018-08-18 11:35:05.927861] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927877] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927885] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927908] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927914] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927936] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-32 >[2018-08-18 11:35:05.927962] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927973] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927986] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927999] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.927986] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.928010] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.928012] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.928025] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.928025] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-19 >[2018-08-18 11:35:05.928034] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-33 >[2018-08-18 11:35:05.928035] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.928053] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.928056] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-34 >[2018-08-18 11:35:05.928091] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.928117] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.928128] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88) from subvol vol0-replicate-35 >[2018-08-18 11:35:05.928140] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-19 cached_subvol null >[2018-08-18 11:35:05.928149] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88 >[2018-08-18 11:35:05.928158] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.928169] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.928180] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.928191] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.928202] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fbf8c424bc0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:05.928489] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-0': avail_percent is: 96.00 and avail_space is: 7682550996992 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.928538] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-1': avail_percent is: 95.00 and avail_space is: 7649217478656 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.928575] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-2': avail_percent is: 96.00 and avail_space is: 7679628898304 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.928630] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-3': avail_percent is: 95.00 and avail_space is: 7676714049536 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.928679] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-4': avail_percent is: 96.00 and avail_space is: 7679852838912 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.928726] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-5': avail_percent is: 95.00 and avail_space is: 7672041328640 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.928790] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-6': avail_percent is: 94.00 and avail_space is: 7567700049920 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.928830] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-7': avail_percent is: 95.00 and avail_space is: 7599561781248 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.928866] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-8': avail_percent is: 95.00 and avail_space is: 7669932085248 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.928919] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-9': avail_percent is: 95.00 and avail_space is: 7675205881856 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.928983] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-10': avail_percent is: 94.00 and avail_space is: 7587254632448 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.929033] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-11': avail_percent is: 95.00 and avail_space is: 7666188136448 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.929076] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-12': avail_percent is: 95.00 and avail_space is: 7662336352256 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.929110] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-13': avail_percent is: 95.00 and avail_space is: 7671819669504 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.929168] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-14': avail_percent is: 95.00 and avail_space is: 7665784659968 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.929208] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-15': avail_percent is: 95.00 and avail_space is: 7654391312384 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.929263] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-16': avail_percent is: 95.00 and avail_space is: 7678806634496 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.929294] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-17': avail_percent is: 95.00 and avail_space is: 7669472206848 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.929345] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-18': avail_percent is: 95.00 and avail_space is: 7615164923904 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.929387] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-19': avail_percent is: 95.00 and avail_space is: 7660373045248 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.929449] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-20': avail_percent is: 95.00 and avail_space is: 7610263240704 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.929487] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-21': avail_percent is: 95.00 and avail_space is: 7663397208064 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.929547] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-22': avail_percent is: 95.00 and avail_space is: 7668074450944 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.929608] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-23': avail_percent is: 95.00 and avail_space is: 7669677629440 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.929695] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-25': avail_percent is: 95.00 and avail_space is: 7644131004416 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.929713] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-24': avail_percent is: 95.00 and avail_space is: 7660807667712 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.929746] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-26': avail_percent is: 95.00 and avail_space is: 7655957213184 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.929796] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-27': avail_percent is: 95.00 and avail_space is: 7652157960192 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.929833] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-28': avail_percent is: 95.00 and avail_space is: 7652197793792 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.929876] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-29': avail_percent is: 95.00 and avail_space is: 7639356854272 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.929925] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-30': avail_percent is: 95.00 and avail_space is: 7675637026816 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.929979] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-31': avail_percent is: 95.00 and avail_space is: 7678164856832 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.930025] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-32': avail_percent is: 95.00 and avail_space is: 7659078995968 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.930077] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-33': avail_percent is: 95.00 and avail_space is: 7676538961920 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.930108] D [MSGID: 0] [dht-common.c:6923:dht_create_wind_to_avail_subvol] 0-vol0-dht: creating /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88 on vol0-replicate-19 >[2018-08-18 11:35:05.930123] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-34': avail_percent is: 95.00 and avail_space is: 7665343188992 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.930170] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-35': avail_percent is: 95.00 and avail_space is: 7661850447872 and avail_inodes is: 99.00 >[2018-08-18 11:35:05.930302] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-19: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:05.960201] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-19 >[2018-08-18 11:35:05.960245] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 11:35:06.069456] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7fbfa80274d0) as it exceeds the maximum available buffer size >[2018-08-18 11:35:06.070823] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.070856] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=d892497a-05eb-456d-9a2a-01f01dab60ea, gen=0): picking the request for winding >[2018-08-18 11:35:06.070875] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=d892497a-05eb-456d-9a2a-01f01dab60ea, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:06.071200] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:06.088394] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=d892497a-05eb-456d-9a2a-01f01dab60ea, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:06.088621] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=d892497a-05eb-456d-9a2a-01f01dab60ea, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:06.088834] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.098624] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7fbfa8022bf0) as it exceeds the maximum available buffer size >[2018-08-18 11:35:06.099845] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.099876] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=1418b68a-5011-45aa-97f0-bf31568ca77f, gen=0): picking the request for winding >[2018-08-18 11:35:06.099893] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=1418b68a-5011-45aa-97f0-bf31568ca77f, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:06.100159] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:06.117001] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=1418b68a-5011-45aa-97f0-bf31568ca77f, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:06.117221] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=1418b68a-5011-45aa-97f0-bf31568ca77f, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:06.117415] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.184775] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7fbfc406a1f0) as it exceeds the maximum available buffer size >[2018-08-18 11:35:06.186217] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.186256] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=7e1c387b-5d37-49ed-8926-eb793fa8192d, gen=0): picking the request for winding >[2018-08-18 11:35:06.186278] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=7e1c387b-5d37-49ed-8926-eb793fa8192d, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:06.186581] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-19: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:06.204437] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=7e1c387b-5d37-49ed-8926-eb793fa8192d, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:06.204555] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=7e1c387b-5d37-49ed-8926-eb793fa8192d, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:06.204669] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.306943] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7fbfa0015d20) as it exceeds the maximum available buffer size >[2018-08-18 11:35:06.308248] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.308282] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=1418b68a-5011-45aa-97f0-bf31568ca77f, gen=0): picking the request for winding >[2018-08-18 11:35:06.308301] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=1418b68a-5011-45aa-97f0-bf31568ca77f, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:06.308366] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:06.308385] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 11:35:06.308416] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7fbfa8022bf0) allocated with standard calloc() >[2018-08-18 11:35:06.322439] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.322480] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=1, gfid=1418b68a-5011-45aa-97f0-bf31568ca77f. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=1418b68a-5011-45aa-97f0-bf31568ca77f, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:06.322934] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=1418b68a-5011-45aa-97f0-bf31568ca77f, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:06.323108] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=1418b68a-5011-45aa-97f0-bf31568ca77f, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:06.323266] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.323288] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=1418b68a-5011-45aa-97f0-bf31568ca77f, gen=1): picking the request for winding >[2018-08-18 11:35:06.323646] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=1418b68a-5011-45aa-97f0-bf31568ca77f, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:06.323762] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 11:35:06.323968] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7fbfa0015d20) allocated with standard calloc() >[2018-08-18 11:35:06.324709] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004 on vol0-replicate-35 >[2018-08-18 11:35:06.324987] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.324983] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325061] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325080] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325094] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004 with op_ret -1 [No such file or directory] >[2018-08-18 11:35:06.325106] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004 missing on subvol vol0-replicate-35 >[2018-08-18 11:35:06.325116] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:06.325412] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325430] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325454] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325509] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325514] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325540] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325516] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325595] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325591] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325590] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-0 >[2018-08-18 11:35:06.325585] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325608] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325650] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-1 >[2018-08-18 11:35:06.325675] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325684] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325693] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325700] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325719] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-2 >[2018-08-18 11:35:06.325714] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325746] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325755] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325789] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325786] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325814] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325816] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-4 >[2018-08-18 11:35:06.325845] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-3 >[2018-08-18 11:35:06.325872] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325873] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325908] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325922] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-5 >[2018-08-18 11:35:06.325971] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325977] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.325973] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326029] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326034] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326047] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-6 >[2018-08-18 11:35:06.326044] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326086] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326112] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326113] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326145] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-7 >[2018-08-18 11:35:06.326181] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326184] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326221] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326236] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326246] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326238] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-8 >[2018-08-18 11:35:06.326280] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326299] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326310] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-9 >[2018-08-18 11:35:06.326335] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326357] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326361] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326381] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326391] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-10 >[2018-08-18 11:35:06.326431] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326465] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326466] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326513] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326518] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326531] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-11 >[2018-08-18 11:35:06.326552] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326579] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326615] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326628] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326622] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326652] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-12 >[2018-08-18 11:35:06.326654] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326675] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326684] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326719] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-13 >[2018-08-18 11:35:06.326723] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326726] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326773] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326792] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-14 >[2018-08-18 11:35:06.326780] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326832] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326839] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326861] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326870] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-15 >[2018-08-18 11:35:06.326926] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326935] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326956] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326976] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.326984] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-16 >[2018-08-18 11:35:06.326990] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327038] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327047] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327081] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327096] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327114] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327116] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-17 >[2018-08-18 11:35:06.327147] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327175] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327183] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327182] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327224] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327225] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-18 >[2018-08-18 11:35:06.327241] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327249] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327267] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-19 >[2018-08-18 11:35:06.327278] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327336] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327348] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327348] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327363] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-20 >[2018-08-18 11:35:06.327376] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327456] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327462] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327474] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327486] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-21 >[2018-08-18 11:35:06.327495] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327489] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327531] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327535] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327558] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327559] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327557] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-22 >[2018-08-18 11:35:06.327607] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327631] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327634] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-23 >[2018-08-18 11:35:06.327635] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327675] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327687] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327694] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-24 >[2018-08-18 11:35:06.327747] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327781] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327787] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327815] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327830] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-25 >[2018-08-18 11:35:06.327861] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327870] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327872] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327908] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327915] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327921] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-26 >[2018-08-18 11:35:06.327953] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327976] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327994] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.327999] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328015] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-27 >[2018-08-18 11:35:06.328042] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328081] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328075] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328101] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328107] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328125] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-28 >[2018-08-18 11:35:06.328133] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328168] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328185] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328199] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-29 >[2018-08-18 11:35:06.328224] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328269] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328279] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328282] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328305] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328308] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328333] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-30 >[2018-08-18 11:35:06.328338] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328358] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328386] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328383] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-31 >[2018-08-18 11:35:06.328383] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328426] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328432] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328473] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328475] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-32 >[2018-08-18 11:35:06.328481] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328522] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328526] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328528] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328536] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-33 >[2018-08-18 11:35:06.328523] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328574] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328588] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328589] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-34 >[2018-08-18 11:35:06.328629] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328634] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328659] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2029970, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328670] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-35 >[2018-08-18 11:35:06.328681] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-35 cached_subvol null >[2018-08-18 11:35:06.328690] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004 >[2018-08-18 11:35:06.328698] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x2029970, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328708] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328719] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328737] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.328747] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x2029970, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.329398] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m returned with op_ret 0 >[2018-08-18 11:35:06.333547] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.333581] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m returned with op_ret 0" repeated 35 times between [2018-08-18 11:35:06.329398] and [2018-08-18 11:35:06.333140] >[2018-08-18 11:35:06.333580] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=1418b68a-5011-45aa-97f0-bf31568ca77f, gen=0): picking the request for winding >[2018-08-18 11:35:06.333647] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88 at vol0-replicate-13 >[2018-08-18 11:35:06.333923] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=1418b68a-5011-45aa-97f0-bf31568ca77f, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:06.333973] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88 returned with op_ret 0 >[2018-08-18 11:35:06.334047] D [md-cache.c:503:mdc_inode_iatt_set_validate] (-->/usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3cd0) [0x7fc07374dcd0] -->/usr/lib64/glusterfs/3.10.12/xlator/performance/md-cache.so(+0xa802) [0x7fc073334802] -->/usr/lib64/glusterfs/3.10.12/xlator/performance/md-cache.so(+0x66cb) [0x7fc0733306cb] ) 0-vol0-md-cache: discarding the iatt validate request(ctime_nsec) [Invalid argument] >[2018-08-18 11:35:06.334223] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004 on vol0-replicate-35 >[2018-08-18 11:35:06.334454] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.334482] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.334482] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.334514] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.334524] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004 with op_ret -1 [No such file or directory] >[2018-08-18 11:35:06.334531] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004 missing on subvol vol0-replicate-35 >[2018-08-18 11:35:06.334538] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:06.334808] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.334815] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.334845] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.334878] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.334899] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.334907] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.334900] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-0 >[2018-08-18 11:35:06.334928] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.334975] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.334979] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.334982] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335012] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-1 >[2018-08-18 11:35:06.335013] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335047] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335048] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335056] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335054] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335079] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-2 >[2018-08-18 11:35:06.335098] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335099] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335115] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335121] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335118] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-3 >[2018-08-18 11:35:06.335142] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335152] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-4 >[2018-08-18 11:35:06.335177] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335185] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335195] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335215] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335226] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-5 >[2018-08-18 11:35:06.335268] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335262] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335278] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335302] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335313] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335317] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335341] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-6 >[2018-08-18 11:35:06.335348] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335367] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335374] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335377] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335378] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-7 >[2018-08-18 11:35:06.335412] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335424] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335446] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335447] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335450] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335473] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-8 >[2018-08-18 11:35:06.335480] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335484] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335511] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-9 >[2018-08-18 11:35:06.335513] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335511] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335539] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335544] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335549] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-10 >[2018-08-18 11:35:06.335569] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335572] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335592] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335617] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335618] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-11 >[2018-08-18 11:35:06.335641] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335653] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335670] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335680] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-12 >[2018-08-18 11:35:06.335692] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335724] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335753] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335768] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335777] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-13 >[2018-08-18 11:35:06.335793] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335803] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335827] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335850] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335864] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-14 >[2018-08-18 11:35:06.335887] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335889] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335896] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335919] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335927] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-15 >[2018-08-18 11:35:06.335957] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335963] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335989] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.335995] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336008] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336036] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336058] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336061] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336065] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336088] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-16 >[2018-08-18 11:35:06.336090] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336100] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336119] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-17 >[2018-08-18 11:35:06.336125] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336139] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336149] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336157] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-18 >[2018-08-18 11:35:06.336155] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336180] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336189] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-19 >[2018-08-18 11:35:06.336228] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336229] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336263] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336266] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336273] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336296] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336303] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336308] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-21 >[2018-08-18 11:35:06.336319] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336333] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-20 >[2018-08-18 11:35:06.336339] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336336] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336346] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336377] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336391] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-22 >[2018-08-18 11:35:06.336412] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336419] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336453] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336472] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336487] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336517] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336519] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336536] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336554] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336555] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336559] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-24 >[2018-08-18 11:35:06.336555] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336577] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-23 >[2018-08-18 11:35:06.336592] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336626] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-25 >[2018-08-18 11:35:06.336628] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336636] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336627] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336659] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336680] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-26 >[2018-08-18 11:35:06.336702] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336704] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336708] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336736] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336765] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336776] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336763] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336808] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336808] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-27 >[2018-08-18 11:35:06.336809] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336809] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336853] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336842] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-28 >[2018-08-18 11:35:06.336882] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336905] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336897] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-29 >[2018-08-18 11:35:06.336907] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336943] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336961] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336966] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336976] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.336975] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-30 >[2018-08-18 11:35:06.337016] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337048] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337050] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337066] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337063] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337108] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337111] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337121] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337108] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-31 >[2018-08-18 11:35:06.337123] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337132] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-32 >[2018-08-18 11:35:06.337165] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337171] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337181] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-33 >[2018-08-18 11:35:06.337189] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337189] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337222] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337243] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337244] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337247] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337267] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-34 >[2018-08-18 11:35:06.337280] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337290] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004) from subvol vol0-replicate-35 >[2018-08-18 11:35:06.337299] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-35 cached_subvol null >[2018-08-18 11:35:06.337306] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004 >[2018-08-18 11:35:06.337312] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337319] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337342] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337350] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337357] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa00187e0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.337481] I [MSGID: 109066] [dht-rename.c:1610:dht_rename] 0-vol0-dht: renaming /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004_c02df208a2da11e8a3196c92bf5c1b88 (hash=vol0-replicate-13/cache=vol0-replicate-13) => /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004 (hash=vol0-replicate-35/cache=<nul>) >[2018-08-18 11:35:06.338172] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-35: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:06.338584] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 11:35:06.338679] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-13: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:06.339006] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 11:35:06.339172] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-35: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:06.339647] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 11:35:06.339755] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-13: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:06.340060] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 11:35:06.340213] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-35: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:06.340423] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 11:35:06.340836] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.340867] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=1418b68a-5011-45aa-97f0-bf31568ca77f, gen=0): picking the request for winding >[2018-08-18 11:35:06.340899] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004 at vol0-replicate-13 >[2018-08-18 11:35:06.341186] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=1418b68a-5011-45aa-97f0-bf31568ca77f, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:06.341211] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004 returned with op_ret 0 >[2018-08-18 11:35:06.341480] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:06.341681] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 11:35:06.341956] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.341982] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=1418b68a-5011-45aa-97f0-bf31568ca77f, gen=0): picking the request for winding >[2018-08-18 11:35:06.342014] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004 at vol0-replicate-13 >[2018-08-18 11:35:06.342301] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=1418b68a-5011-45aa-97f0-bf31568ca77f, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:06.342333] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004 returned with op_ret 0 >[2018-08-18 11:35:06.342591] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:06.342798] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 11:35:06.343072] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.343098] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=1418b68a-5011-45aa-97f0-bf31568ca77f, gen=0): picking the request for winding >[2018-08-18 11:35:06.343130] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004 at vol0-replicate-13 >[2018-08-18 11:35:06.343401] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=1418b68a-5011-45aa-97f0-bf31568ca77f, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:06.343451] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004 returned with op_ret 0 >[2018-08-18 11:35:06.343748] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:06.343938] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 11:35:06.344210] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.344236] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=1418b68a-5011-45aa-97f0-bf31568ca77f, gen=0): picking the request for winding >[2018-08-18 11:35:06.344273] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004 at vol0-replicate-13 >[2018-08-18 11:35:06.344543] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=1418b68a-5011-45aa-97f0-bf31568ca77f, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:06.344608] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0004 returned with op_ret 0 >[2018-08-18 11:35:06.344873] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:06.345100] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 11:35:06.362431] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7fbf9c00fcd0) as it exceeds the maximum available buffer size >[2018-08-18 11:35:06.363706] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.363738] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=7e1c387b-5d37-49ed-8926-eb793fa8192d, gen=0): picking the request for winding >[2018-08-18 11:35:06.363757] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=7e1c387b-5d37-49ed-8926-eb793fa8192d, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:06.363827] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-19: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:06.363847] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 11:35:06.363877] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7fbfc406a1f0) allocated with standard calloc() >[2018-08-18 11:35:06.375099] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.375137] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=1, gfid=7e1c387b-5d37-49ed-8926-eb793fa8192d. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=7e1c387b-5d37-49ed-8926-eb793fa8192d, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:06.379111] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=7e1c387b-5d37-49ed-8926-eb793fa8192d, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:06.379231] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=7e1c387b-5d37-49ed-8926-eb793fa8192d, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:06.379333] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.379348] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=7e1c387b-5d37-49ed-8926-eb793fa8192d, gen=1): picking the request for winding >[2018-08-18 11:35:06.379543] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=7e1c387b-5d37-49ed-8926-eb793fa8192d, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:06.379840] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 11:35:06.379992] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7fbf9c00fcd0) allocated with standard calloc() >[2018-08-18 11:35:06.381155] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007 on vol0-replicate-18 >[2018-08-18 11:35:06.381477] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.381493] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.381527] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.381559] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.381585] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007 with op_ret -1 [No such file or directory] >[2018-08-18 11:35:06.381611] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007 missing on subvol vol0-replicate-18 >[2018-08-18 11:35:06.381627] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:06.381932] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.381969] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.381969] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382008] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382019] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-0 >[2018-08-18 11:35:06.382050] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382051] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382075] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382097] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382107] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-1 >[2018-08-18 11:35:06.382129] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382152] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382166] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382180] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382189] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-2 >[2018-08-18 11:35:06.382254] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382282] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382303] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382335] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382342] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382347] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382363] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-3 >[2018-08-18 11:35:06.382387] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382422] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382451] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382446] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-4 >[2018-08-18 11:35:06.382464] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382492] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382527] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382538] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382542] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-5 >[2018-08-18 11:35:06.382542] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382571] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382582] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382596] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-6 >[2018-08-18 11:35:06.382624] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382630] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382655] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382667] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382677] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-7 >[2018-08-18 11:35:06.382719] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382735] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382743] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382761] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382771] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-8 >[2018-08-18 11:35:06.382830] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382870] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382911] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382932] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382945] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-9 >[2018-08-18 11:35:06.382955] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.382956] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383016] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383040] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383045] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383049] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383061] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383075] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-10 >[2018-08-18 11:35:06.383081] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383092] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-11 >[2018-08-18 11:35:06.383109] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383156] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383162] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383197] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383211] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-12 >[2018-08-18 11:35:06.383239] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383249] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383262] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383278] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383287] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-13 >[2018-08-18 11:35:06.383390] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383397] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383417] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383430] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383438] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-14 >[2018-08-18 11:35:06.383471] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383486] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383500] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383514] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383521] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-15 >[2018-08-18 11:35:06.383569] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383584] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383607] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383628] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383636] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-16 >[2018-08-18 11:35:06.383650] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383670] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383680] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383700] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383710] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-17 >[2018-08-18 11:35:06.383707] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383738] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383780] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383793] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383815] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383832] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383843] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-19 >[2018-08-18 11:35:06.383931] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383930] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383947] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383977] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.383988] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-20 >[2018-08-18 11:35:06.384019] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384030] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384071] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384087] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384097] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-21 >[2018-08-18 11:35:06.384130] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384155] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384181] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384192] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384200] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-22 >[2018-08-18 11:35:06.384264] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384274] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384270] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384300] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384314] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-23 >[2018-08-18 11:35:06.384340] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384377] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384387] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384401] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384410] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-24 >[2018-08-18 11:35:06.384470] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384479] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384468] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384527] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384543] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384553] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-25 >[2018-08-18 11:35:06.384573] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384580] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384607] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384624] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-26 >[2018-08-18 11:35:06.384645] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384668] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384700] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384722] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384735] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-27 >[2018-08-18 11:35:06.384768] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384791] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384861] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384875] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384887] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-28 >[2018-08-18 11:35:06.384897] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384903] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384922] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384934] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.384942] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-29 >[2018-08-18 11:35:06.385000] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385014] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385043] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385057] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385067] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-30 >[2018-08-18 11:35:06.385085] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385084] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385130] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385143] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385151] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-31 >[2018-08-18 11:35:06.385218] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385224] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385245] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385263] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385278] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-32 >[2018-08-18 11:35:06.385307] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385328] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385328] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385347] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385354] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-33 >[2018-08-18 11:35:06.385360] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385384] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385386] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385400] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-18 >[2018-08-18 11:35:06.385428] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385475] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385496] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385500] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385519] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385519] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385545] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-35 >[2018-08-18 11:35:06.385545] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x2136b70, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385573] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-34 >[2018-08-18 11:35:06.385587] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-18 cached_subvol null >[2018-08-18 11:35:06.385606] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007 >[2018-08-18 11:35:06.385621] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x2136b70, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385633] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385645] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385658] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.385670] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x2136b70, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.386330] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m returned with op_ret 0 >[2018-08-18 11:35:06.390708] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.390749] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m returned with op_ret 0" repeated 35 times between [2018-08-18 11:35:06.386330] and [2018-08-18 11:35:06.390219] >[2018-08-18 11:35:06.390747] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=7e1c387b-5d37-49ed-8926-eb793fa8192d, gen=0): picking the request for winding >[2018-08-18 11:35:06.390812] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88 at vol0-replicate-19 >[2018-08-18 11:35:06.391137] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88 returned with op_ret 0 >[2018-08-18 11:35:06.391144] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=7e1c387b-5d37-49ed-8926-eb793fa8192d, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:06.391265] D [md-cache.c:493:mdc_inode_iatt_set_validate] (-->/usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3cd0) [0x7fc07374dcd0] -->/usr/lib64/glusterfs/3.10.12/xlator/performance/md-cache.so(+0xa802) [0x7fc073334802] -->/usr/lib64/glusterfs/3.10.12/xlator/performance/md-cache.so(+0x66cb) [0x7fc0733306cb] ) 0-vol0-md-cache: discarding the iatt validate request [Invalid argument] >[2018-08-18 11:35:06.391436] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007 on vol0-replicate-18 >[2018-08-18 11:35:06.391690] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.391734] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.391729] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.391766] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.391780] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007 with op_ret -1 [No such file or directory] >[2018-08-18 11:35:06.391793] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007 missing on subvol vol0-replicate-18 >[2018-08-18 11:35:06.391803] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:06.392040] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392062] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392076] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392090] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392098] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-0 >[2018-08-18 11:35:06.392128] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392156] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392157] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392190] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392204] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-1 >[2018-08-18 11:35:06.392205] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392232] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392249] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392268] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392279] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-2 >[2018-08-18 11:35:06.392296] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392313] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392334] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392348] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392363] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-3 >[2018-08-18 11:35:06.392368] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392405] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392408] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392429] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392438] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-4 >[2018-08-18 11:35:06.392454] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392471] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392495] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392511] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392522] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-5 >[2018-08-18 11:35:06.392554] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392555] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392555] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392585] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392594] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-6 >[2018-08-18 11:35:06.392616] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392628] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392642] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392654] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392661] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-7 >[2018-08-18 11:35:06.392689] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392706] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392732] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392764] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392776] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392779] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-8 >[2018-08-18 11:35:06.392802] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392844] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392858] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392871] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392890] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392890] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-9 >[2018-08-18 11:35:06.392902] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392922] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392934] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-10 >[2018-08-18 11:35:06.392949] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.392967] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393017] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393039] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393049] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393056] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-11 >[2018-08-18 11:35:06.393064] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393096] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393087] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393120] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393125] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393145] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393146] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-12 >[2018-08-18 11:35:06.393159] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393169] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-13 >[2018-08-18 11:35:06.393180] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393189] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393214] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393230] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393242] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-14 >[2018-08-18 11:35:06.393294] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393313] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393317] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393335] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393350] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-15 >[2018-08-18 11:35:06.393377] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393396] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393408] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393423] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393430] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-16 >[2018-08-18 11:35:06.393447] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393472] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393476] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393496] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393504] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-17 >[2018-08-18 11:35:06.393545] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393536] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393581] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393595] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393616] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393628] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393636] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-19 >[2018-08-18 11:35:06.393660] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393667] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393694] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393710] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393720] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-20 >[2018-08-18 11:35:06.393766] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393768] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393789] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393802] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393810] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-21 >[2018-08-18 11:35:06.393842] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393894] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393928] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393944] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393942] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393976] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393991] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.393983] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394000] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-22 >[2018-08-18 11:35:06.394025] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394033] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394049] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394064] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394058] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394074] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394057] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-23 >[2018-08-18 11:35:06.394087] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-24 >[2018-08-18 11:35:06.394106] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394117] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394125] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-25 >[2018-08-18 11:35:06.394128] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394129] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394150] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394163] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394170] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-26 >[2018-08-18 11:35:06.394206] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394210] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394231] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394248] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394258] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-27 >[2018-08-18 11:35:06.394286] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394289] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394301] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394320] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394333] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-28 >[2018-08-18 11:35:06.394353] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394357] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394372] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394384] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394391] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-29 >[2018-08-18 11:35:06.394418] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394446] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394498] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394527] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394529] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394539] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394546] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394560] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394561] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-31 >[2018-08-18 11:35:06.394578] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-30 >[2018-08-18 11:35:06.394573] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394588] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394589] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394632] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394634] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394643] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394649] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-32 >[2018-08-18 11:35:06.394650] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394668] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394675] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-33 >[2018-08-18 11:35:06.394686] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394699] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394714] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394717] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394718] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394738] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394739] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-18 >[2018-08-18 11:35:06.394750] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-34 >[2018-08-18 11:35:06.394791] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394795] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394798] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394818] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394825] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007) from subvol vol0-replicate-35 >[2018-08-18 11:35:06.394834] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-18 cached_subvol null >[2018-08-18 11:35:06.394841] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007 >[2018-08-18 11:35:06.394847] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394854] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394861] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394868] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.394884] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfc40416f0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.395006] I [MSGID: 109066] [dht-rename.c:1610:dht_rename] 0-vol0-dht: renaming /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007_c02f1e44a2da11e8b9cb6c92bf5c1b88 (hash=vol0-replicate-19/cache=vol0-replicate-19) => /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007 (hash=vol0-replicate-18/cache=<nul>) >[2018-08-18 11:35:06.395687] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-18: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:06.425037] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 11:35:06.425140] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-19: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:06.425506] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 11:35:06.425635] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-18: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:06.426088] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 11:35:06.426224] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-19: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:06.426486] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 11:35:06.426555] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-18: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:06.426740] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 11:35:06.427260] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.427283] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=7e1c387b-5d37-49ed-8926-eb793fa8192d, gen=0): picking the request for winding >[2018-08-18 11:35:06.427313] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007 at vol0-replicate-19 >[2018-08-18 11:35:06.427557] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=7e1c387b-5d37-49ed-8926-eb793fa8192d, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:06.427694] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007 returned with op_ret 0 >[2018-08-18 11:35:06.427934] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-19: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:06.428111] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 11:35:06.428330] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.428350] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=7e1c387b-5d37-49ed-8926-eb793fa8192d, gen=0): picking the request for winding >[2018-08-18 11:35:06.428374] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007 at vol0-replicate-19 >[2018-08-18 11:35:06.428594] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=7e1c387b-5d37-49ed-8926-eb793fa8192d, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:06.428684] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007 returned with op_ret 0 >[2018-08-18 11:35:06.428910] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-19: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:06.429181] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 11:35:06.429382] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.429400] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=7e1c387b-5d37-49ed-8926-eb793fa8192d, gen=0): picking the request for winding >[2018-08-18 11:35:06.429423] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007 at vol0-replicate-19 >[2018-08-18 11:35:06.429633] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=7e1c387b-5d37-49ed-8926-eb793fa8192d, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:06.429690] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007 returned with op_ret 0 >[2018-08-18 11:35:06.429907] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-19: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:06.430087] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 11:35:06.430288] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.430306] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=7e1c387b-5d37-49ed-8926-eb793fa8192d, gen=0): picking the request for winding >[2018-08-18 11:35:06.430329] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007 at vol0-replicate-19 >[2018-08-18 11:35:06.430530] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=7e1c387b-5d37-49ed-8926-eb793fa8192d, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:06.430572] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7fbf9c0b8de0) as it exceeds the maximum available buffer size >[2018-08-18 11:35:06.430610] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/f6275d53e1cf343d90b28025b919da97_171654950869273281.m/0007 returned with op_ret 0 >[2018-08-18 11:35:06.430864] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-19: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:06.431103] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 11:35:06.431712] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.431746] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=d892497a-05eb-456d-9a2a-01f01dab60ea, gen=0): picking the request for winding >[2018-08-18 11:35:06.431764] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=d892497a-05eb-456d-9a2a-01f01dab60ea, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:06.431844] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:06.431868] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:06.431900] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7fbfa80274d0) allocated with standard calloc() >[2018-08-18 11:35:06.444930] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.444970] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=1, gfid=d892497a-05eb-456d-9a2a-01f01dab60ea. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=d892497a-05eb-456d-9a2a-01f01dab60ea, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:06.447852] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=d892497a-05eb-456d-9a2a-01f01dab60ea, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:06.447970] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=d892497a-05eb-456d-9a2a-01f01dab60ea, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:06.448071] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.448086] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=d892497a-05eb-456d-9a2a-01f01dab60ea, gen=1): picking the request for winding >[2018-08-18 11:35:06.448292] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=d892497a-05eb-456d-9a2a-01f01dab60ea, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:06.448463] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:06.448656] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7fbf9c0b8de0) allocated with standard calloc() >[2018-08-18 11:35:06.449066] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003 on vol0-replicate-4 >[2018-08-18 11:35:06.449328] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.449340] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.449361] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.449383] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.449394] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003 with op_ret -1 [No such file or directory] >[2018-08-18 11:35:06.449403] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003 missing on subvol vol0-replicate-4 >[2018-08-18 11:35:06.449410] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:06.449675] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.449678] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.449719] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.449731] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.449732] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.449764] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.449766] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.449781] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-0 >[2018-08-18 11:35:06.449797] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.449809] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-1 >[2018-08-18 11:35:06.449829] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.449837] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.449830] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.449871] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.449886] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-2 >[2018-08-18 11:35:06.449912] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.449942] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.449946] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.449950] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.449950] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.449982] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.449996] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-3 >[2018-08-18 11:35:06.450043] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450049] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450065] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450081] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450091] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-5 >[2018-08-18 11:35:06.450129] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450133] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450165] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450190] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450196] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450204] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-6 >[2018-08-18 11:35:06.450199] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450229] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450250] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450254] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450259] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-7 >[2018-08-18 11:35:06.450273] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450282] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450297] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450309] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-8 >[2018-08-18 11:35:06.450317] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450310] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450354] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450373] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450383] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-9 >[2018-08-18 11:35:06.450386] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450410] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450407] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450433] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450444] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-10 >[2018-08-18 11:35:06.450476] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450488] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450497] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450508] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450515] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-11 >[2018-08-18 11:35:06.450563] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450580] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450610] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450632] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450633] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450657] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-12 >[2018-08-18 11:35:06.450662] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450662] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450694] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450717] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450713] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450696] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450741] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450742] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-13 >[2018-08-18 11:35:06.450753] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-14 >[2018-08-18 11:35:06.450770] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450795] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450814] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450838] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450842] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450861] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-15 >[2018-08-18 11:35:06.450866] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450882] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450897] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450906] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-16 >[2018-08-18 11:35:06.450909] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450930] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450958] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450969] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450976] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-17 >[2018-08-18 11:35:06.450991] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450992] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.450999] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451033] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451044] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-18 >[2018-08-18 11:35:06.451051] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451055] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451077] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451102] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451105] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451117] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-19 >[2018-08-18 11:35:06.451129] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451145] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451160] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451173] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-20 >[2018-08-18 11:35:06.451226] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451241] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451243] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451259] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451267] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-21 >[2018-08-18 11:35:06.451276] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451287] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451302] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451312] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451319] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-22 >[2018-08-18 11:35:06.451349] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451377] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451396] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451412] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451425] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451439] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451440] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451461] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-23 >[2018-08-18 11:35:06.451463] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451465] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451486] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-24 >[2018-08-18 11:35:06.451494] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451510] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451523] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451532] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-25 >[2018-08-18 11:35:06.451538] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451557] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451562] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451576] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451583] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-26 >[2018-08-18 11:35:06.451624] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451627] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451632] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451660] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451668] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-27 >[2018-08-18 11:35:06.451688] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451687] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451720] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451739] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451744] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451757] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451761] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-28 >[2018-08-18 11:35:06.451799] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451799] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451817] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451819] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451833] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-29 >[2018-08-18 11:35:06.451884] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451888] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451898] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451912] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451922] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451919] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-31 >[2018-08-18 11:35:06.451936] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451941] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451956] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451959] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451959] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-30 >[2018-08-18 11:35:06.451982] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.451992] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-32 >[2018-08-18 11:35:06.452020] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.452032] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.452034] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.452056] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.452054] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.452068] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-33 >[2018-08-18 11:35:06.452077] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.452082] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.452088] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-4 >[2018-08-18 11:35:06.452099] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.452100] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.452123] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.452139] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-34 >[2018-08-18 11:35:06.452161] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.452168] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.452165] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.452187] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.452197] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-35 >[2018-08-18 11:35:06.452207] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-4 cached_subvol null >[2018-08-18 11:35:06.452215] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003 >[2018-08-18 11:35:06.452223] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x218b2c0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.452231] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.452242] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.452252] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.452262] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x218b2c0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.452926] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m returned with op_ret 0 >[2018-08-18 11:35:06.455812] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.455841] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m returned with op_ret 0" repeated 35 times between [2018-08-18 11:35:06.452926] and [2018-08-18 11:35:06.455447] >[2018-08-18 11:35:06.455840] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=d892497a-05eb-456d-9a2a-01f01dab60ea, gen=0): picking the request for winding >[2018-08-18 11:35:06.455891] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88 at vol0-replicate-11 >[2018-08-18 11:35:06.456139] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=d892497a-05eb-456d-9a2a-01f01dab60ea, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:06.456183] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88 returned with op_ret 0 >[2018-08-18 11:35:06.456287] D [md-cache.c:493:mdc_inode_iatt_set_validate] (-->/usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3cd0) [0x7fc07374dcd0] -->/usr/lib64/glusterfs/3.10.12/xlator/performance/md-cache.so(+0xa802) [0x7fc073334802] -->/usr/lib64/glusterfs/3.10.12/xlator/performance/md-cache.so(+0x66cb) [0x7fc0733306cb] ) 0-vol0-md-cache: discarding the iatt validate request [Invalid argument] >[2018-08-18 11:35:06.456426] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003 on vol0-replicate-4 >[2018-08-18 11:35:06.456644] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.456664] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.456671] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.456686] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.456694] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003 with op_ret -1 [No such file or directory] >[2018-08-18 11:35:06.456702] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003 missing on subvol vol0-replicate-4 >[2018-08-18 11:35:06.456708] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 11:35:06.456930] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.456934] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.456943] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.456975] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.456985] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-0 >[2018-08-18 11:35:06.456995] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.456996] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457002] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457027] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457037] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-1 >[2018-08-18 11:35:06.457042] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457042] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457058] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457073] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457081] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-2 >[2018-08-18 11:35:06.457089] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457090] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457111] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457134] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457141] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457142] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457147] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-3 >[2018-08-18 11:35:06.457221] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457234] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457232] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457263] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457274] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-5 >[2018-08-18 11:35:06.457284] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457285] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457287] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457309] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457316] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-6 >[2018-08-18 11:35:06.457347] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457345] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457344] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457379] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457381] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457382] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457406] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457406] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-7 >[2018-08-18 11:35:06.457423] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457434] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457435] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457454] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457454] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-8 >[2018-08-18 11:35:06.457473] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457484] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-9 >[2018-08-18 11:35:06.457491] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457520] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457559] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457574] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457574] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457603] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457607] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457620] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457622] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457624] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-11 >[2018-08-18 11:35:06.457629] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457646] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-10 >[2018-08-18 11:35:06.457646] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457670] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457678] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-12 >[2018-08-18 11:35:06.457687] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457702] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457718] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457735] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457741] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457747] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-13 >[2018-08-18 11:35:06.457756] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457763] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457776] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457784] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-14 >[2018-08-18 11:35:06.457803] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457814] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457838] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457855] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457871] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-15 >[2018-08-18 11:35:06.457877] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457894] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457900] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457914] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457924] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-16 >[2018-08-18 11:35:06.457932] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457940] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457954] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457968] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.457977] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-17 >[2018-08-18 11:35:06.457997] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458017] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458025] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458046] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458048] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458060] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-18 >[2018-08-18 11:35:06.458073] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458107] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458121] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458126] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458121] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458135] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458146] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458147] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-19 >[2018-08-18 11:35:06.458157] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-20 >[2018-08-18 11:35:06.458157] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458186] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458191] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458207] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458214] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-21 >[2018-08-18 11:35:06.458231] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458243] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458265] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458280] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458289] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-22 >[2018-08-18 11:35:06.458311] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458314] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458332] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458347] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458369] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458368] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458368] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-23 >[2018-08-18 11:35:06.458390] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458409] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458409] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458428] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458428] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-24 >[2018-08-18 11:35:06.458473] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458486] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458488] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458509] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458510] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458528] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458532] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458533] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458533] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-26 >[2018-08-18 11:35:06.458538] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458548] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-25 >[2018-08-18 11:35:06.458567] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458576] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-27 >[2018-08-18 11:35:06.458579] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458585] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458619] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458627] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458638] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458646] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458652] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-28 >[2018-08-18 11:35:06.458678] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458694] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458695] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458708] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-29 >[2018-08-18 11:35:06.458718] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458730] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458743] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458748] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458751] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-30 >[2018-08-18 11:35:06.458774] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458784] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458796] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458803] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-31 >[2018-08-18 11:35:06.458820] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458820] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458826] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458847] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458854] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-32 >[2018-08-18 11:35:06.458875] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458878] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458903] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458909] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458907] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458914] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-4 >[2018-08-18 11:35:06.458925] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458929] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458936] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458945] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-33 >[2018-08-18 11:35:06.458946] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458966] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458973] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-34 >[2018-08-18 11:35:06.458990] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.458993] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.459012] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.459024] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.459031] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003) from subvol vol0-replicate-35 >[2018-08-18 11:35:06.459042] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-4 cached_subvol null >[2018-08-18 11:35:06.459048] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003 >[2018-08-18 11:35:06.459053] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.459060] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.459067] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.459074] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.459081] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fbfa80023d0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 11:35:06.459172] I [MSGID: 109066] [dht-rename.c:1610:dht_rename] 0-vol0-dht: renaming /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003_c07dc71aa2da11e8a8ce6c92bf5c1b88 (hash=vol0-replicate-11/cache=vol0-replicate-11) => /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003 (hash=vol0-replicate-4/cache=<nul>) >[2018-08-18 11:35:06.459746] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-4: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:06.460058] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:06.460144] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-11: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:06.460405] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:06.460517] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-4: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:06.460973] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:06.461045] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-11: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 11:35:06.461306] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:06.461356] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:06.461523] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 11:35:06.462074] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.462108] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=d892497a-05eb-456d-9a2a-01f01dab60ea, gen=0): picking the request for winding >[2018-08-18 11:35:06.462148] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003 at vol0-replicate-11 >[2018-08-18 11:35:06.462449] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003 returned with op_ret 0 >[2018-08-18 11:35:06.462454] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=d892497a-05eb-456d-9a2a-01f01dab60ea, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:06.462769] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:06.462973] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:06.463302] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.463335] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=d892497a-05eb-456d-9a2a-01f01dab60ea, gen=0): picking the request for winding >[2018-08-18 11:35:06.463372] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003 at vol0-replicate-11 >[2018-08-18 11:35:06.463678] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003 returned with op_ret 0 >[2018-08-18 11:35:06.463720] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=d892497a-05eb-456d-9a2a-01f01dab60ea, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:06.464025] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:06.464208] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:06.464528] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.464564] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=d892497a-05eb-456d-9a2a-01f01dab60ea, gen=0): picking the request for winding >[2018-08-18 11:35:06.464613] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003 at vol0-replicate-11 >[2018-08-18 11:35:06.464944] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003 returned with op_ret 0 >[2018-08-18 11:35:06.464943] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=d892497a-05eb-456d-9a2a-01f01dab60ea, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:06.465232] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:06.465420] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:06.465751] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:06.465785] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=d892497a-05eb-456d-9a2a-01f01dab60ea, gen=0): picking the request for winding >[2018-08-18 11:35:06.465822] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003 at vol0-replicate-11 >[2018-08-18 11:35:06.466151] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/22b9a366487b3ea08f7fc40fae4293c9_171654953037755030.m/0003 returned with op_ret 0 >[2018-08-18 11:35:06.466141] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=d892497a-05eb-456d-9a2a-01f01dab60ea, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:06.466438] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:06.466628] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 11:35:07.383606] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:07.383656] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:07.383707] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t at vol0-replicate-24 >[2018-08-18 11:35:07.384076] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:07.384137] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t returned with op_ret 0 >[2018-08-18 11:35:07.384730] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7fc073b634e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7fc082b88631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7fc07374e820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:07.384765] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:07.384802] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: 6135251d-bd90-46da-a007-650da6fbe9d0: generation now vs cached: 3, 3 >[2018-08-18 11:35:07.385049] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:07.385836] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:07.385874] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:07.385896] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:07.386164] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:07.386853] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:07.387043] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:07.387221] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:07.388979] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:07.389012] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:07.389032] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:07.389092] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:07.389111] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:07.390176] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:07.390208] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:07.390230] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:07.390563] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:07.390748] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 11:35:07.390775] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:07.390934] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:07.391229] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:07.391253] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): picking the request for winding >[2018-08-18 11:35:07.391273] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:07.391286] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:07.391501] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:07.391702] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:07.391773] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:07.391790] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:07.391889] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:07.391916] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:07.392139] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] 0-vol0-write-behind: processing queues >[2018-08-18 11:35:07.392202] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:07.392356] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:07.392392] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): picking the request for winding >[2018-08-18 11:35:07.392409] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): picking the request for winding >[2018-08-18 11:35:07.392421] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:07.392473] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:07.392490] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:07.392511] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:07.392593] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:07.393028] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:07.393250] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:07.393521] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:07.393560] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:07.393606] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:07.393679] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 11:35:07.393639] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:07.393678] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:07.393959] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:07.394155] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:07.394318] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:07.394341] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): picking the request for winding >[2018-08-18 11:35:07.394673] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): destroying request, removing from all queues >[2018-08-18 11:35:07.394801] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:09.881640] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:09.881685] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:09.881737] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t at vol0-replicate-24 >[2018-08-18 11:35:09.882098] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:09.882145] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t returned with op_ret 0 >[2018-08-18 11:35:09.882740] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7fc073b634e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7fc082b88631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7fc07374e820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:09.882775] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:09.882811] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: 6135251d-bd90-46da-a007-650da6fbe9d0: generation now vs cached: 3, 3 >[2018-08-18 11:35:09.883063] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:09.883769] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:09.883808] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:09.883831] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:09.884089] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:09.884781] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:09.884950] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:09.885136] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:09.885157] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:09.885172] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:09.885228] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:09.885242] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:09.885313] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:09.885942] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:09.885977] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:09.886000] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:09.886416] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:09.886465] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:09.886627] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:09.886832] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:09.886875] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 11:35:09.886907] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): picking the request for winding >[2018-08-18 11:35:09.887112] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:09.887181] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:09.887202] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:09.887750] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.s(--> /usr/lib64/glusterfs/3(--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:09.887788] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.s(--> /usr/lib64/glusterfs/3(--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:09.887904] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:09.887943] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): picking the request for winding >[2018-08-18 11:35:09.887963] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:09.888022] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:09.888040] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:09.888043] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:09.888517] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:09.888530] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:09.888695] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:09.888734] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): picking the request for winding >[2018-08-18 11:35:09.888754] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:09.888808] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:09.888829] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:09.888839] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:09.889386] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:09.889398] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:09.889546] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:09.889586] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): picking the request for winding >[2018-08-18 11:35:09.889703] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:09.889927] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:09.889941] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): destroying request, removing from all queues >[2018-08-18 11:35:12.886540] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:12.886593] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:12.886674] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t at vol0-replicate-24 >[2018-08-18 11:35:12.887096] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:12.887193] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t returned with op_ret 0 >[2018-08-18 11:35:12.887864] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7fc073b634e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7fc082b88631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7fc07374e820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:12.887899] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:12.887943] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: 6135251d-bd90-46da-a007-650da6fbe9d0: generation now vs cached: 3, 3 >[2018-08-18 11:35:12.888199] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:12.888879] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:12.888921] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:12.888946] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:12.889215] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:12.890020] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:12.890233] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:12.890427] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:12.891978] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:12.892019] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:12.892041] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:12.892101] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:12.892118] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:12.893848] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:12.893901] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:12.893929] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:12.894430] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:12.894682] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 11:35:12.894718] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:12.895010] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:12.895220] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:12.895551] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:12.895572] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:12.895629] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): picking the request for winding >[2018-08-18 11:35:12.895713] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:12.895733] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:12.896013] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:12.896241] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:12.896431] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:12.896485] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): picking the request for winding >[2018-08-18 11:35:12.896514] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:12.896595] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:12.896636] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:12.896652] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:12.897202] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:12.897244] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:12.897434] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:12.897479] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): picking the request for winding >[2018-08-18 11:35:12.897504] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:12.897571] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:12.897610] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:12.897620] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:12.898318] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:12.898336] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:12.898540] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:12.898586] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): picking the request for winding >[2018-08-18 11:35:12.898729] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:12.898981] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:12.898989] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): destroying request, removing from all queues >[2018-08-18 11:35:15.948118] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:15.948165] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:15.948223] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t at vol0-replicate-24 >[2018-08-18 11:35:15.948647] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:15.948763] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t returned with op_ret 0 >[2018-08-18 11:35:15.949413] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7fc073b634e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7fc082b88631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7fc07374e820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:15.949449] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:15.949491] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: 6135251d-bd90-46da-a007-650da6fbe9d0: generation now vs cached: 3, 3 >[2018-08-18 11:35:15.949781] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:15.950629] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:15.950676] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:15.950709] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:15.951011] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:15.951908] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:15.952153] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:15.952370] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:15.953775] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:15.953813] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:15.953836] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:15.953907] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:15.953930] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:15.955471] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:15.955520] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:15.955546] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:15.956106] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:15.956322] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 11:35:15.956352] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:15.956673] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:15.956910] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:15.957225] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:15.957263] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:15.957315] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): picking the request for winding >[2018-08-18 11:35:15.957385] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:15.957406] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:15.957734] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:15.958029] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:15.958265] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:15.958317] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): picking the request for winding >[2018-08-18 11:35:15.958342] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:15.958413] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:15.958433] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:15.958489] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:15.959001] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:15.959042] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:15.959261] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:15.959308] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): picking the request for winding >[2018-08-18 11:35:15.959331] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:15.959398] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:15.959417] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:15.959464] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:15.959950] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:15.960164] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:15.960506] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:15.960547] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:15.960632] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:15.961052] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:15.961225] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:19.053814] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:19.053855] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:19.053902] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t at vol0-replicate-24 >[2018-08-18 11:35:19.054249] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:19.054907] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7fc073b634e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7fc082b88631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7fc07374e820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:19.054940] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 11:35:19.054381] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t returned with op_ret 0 >[2018-08-18 11:35:19.054939] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:19.054999] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: 6135251d-bd90-46da-a007-650da6fbe9d0: generation now vs cached: 3, 3 >[2018-08-18 11:35:19.055227] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:19.056027] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:19.056073] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:19.056098] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:19.056371] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:19.057139] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:19.057364] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:19.057568] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:19.059428] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:19.059461] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:19.059482] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:19.059548] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:19.059567] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:19.060855] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:19.060895] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:19.060916] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:19.061404] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:19.061622] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 11:35:19.061658] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:19.061846] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:19.062239] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:19.062374] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:19.062554] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:19.062594] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): picking the request for winding >[2018-08-18 11:35:19.062623] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): picking the request for winding >[2018-08-18 11:35:19.062637] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:19.062696] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:19.062716] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:19.062732] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:19.062849] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:19.063197] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:19.063393] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:19.063751] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:19.063762] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:19.063793] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:19.063812] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:19.063834] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:19.063911] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:19.064120] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:19.064299] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:19.064651] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:19.064661] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:19.064692] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): picking the request for winding >[2018-08-18 11:35:19.064716] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:19.064767] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 11:35:19.064744] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:19.064767] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:19.064801] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:19.065196] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:19.065370] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:19.065534] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:19.065558] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): picking the request for winding >[2018-08-18 11:35:19.065882] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): destroying request, removing from all queues >[2018-08-18 11:35:19.066004] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:21.777917] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:21.777963] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:21.778015] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t at vol0-replicate-24 >[2018-08-18 11:35:21.778393] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:21.778514] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t returned with op_ret 0 >[2018-08-18 11:35:21.779106] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7fc073b634e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7fc082b88631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7fc07374e820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:21.779142] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:21.779178] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: 6135251d-bd90-46da-a007-650da6fbe9d0: generation now vs cached: 3, 3 >[2018-08-18 11:35:21.779423] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:21.780134] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:21.780168] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:21.780189] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:21.780444] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:21.781116] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:21.781292] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:21.781460] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:21.782450] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:21.782485] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:21.782506] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:21.782564] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:21.782583] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:21.783485] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:21.783510] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:21.783524] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:21.783807] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:21.783940] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 11:35:21.783963] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:21.784119] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:21.784407] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:21.784503] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:21.784681] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:21.784721] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): picking the request for winding >[2018-08-18 11:35:21.784737] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): picking the request for winding >[2018-08-18 11:35:21.784748] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:21.784792] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:21.784806] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:21.784842] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:21.784914] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:21.785215] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:21.785348] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:21.785574] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:21.785639] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:21.785655] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:21.785835] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:21.785866] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): picking the request for winding >[2018-08-18 11:35:21.785895] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:21.785955] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:21.785967] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:21.785976] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:21.785998] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:21.786398] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:21.786596] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:21.786800] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:21.786833] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:21.786912] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:21.787133] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:21.787320] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:21.787507] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:21.787534] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): picking the request for winding >[2018-08-18 11:35:21.787904] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): destroying request, removing from all queues >[2018-08-18 11:35:21.788036] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:24.637891] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:24.637938] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:24.637988] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t at vol0-replicate-24 >[2018-08-18 11:35:24.638358] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:24.638461] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t returned with op_ret 0 >[2018-08-18 11:35:24.639084] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7fc073b634e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7fc082b88631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7fc07374e820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:24.639125] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:24.639162] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: 6135251d-bd90-46da-a007-650da6fbe9d0: generation now vs cached: 3, 3 >[2018-08-18 11:35:24.639415] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:24.640180] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:24.640223] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:24.640247] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:24.640535] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:24.641325] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:24.641548] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:24.641760] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:24.643151] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:24.643187] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:24.643208] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:24.643267] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:24.643288] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:24.644395] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:24.644429] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:24.644449] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:24.644868] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:24.645079] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 11:35:24.645109] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:24.645299] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:24.645642] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:24.645853] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:24.646116] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:24.646146] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): picking the request for winding >[2018-08-18 11:35:24.646170] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): picking the request for winding >[2018-08-18 11:35:24.646187] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:24.646199] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:24.646252] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:24.646274] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:24.646415] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:24.646511] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:24.646539] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:24.646559] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:24.646838] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:24.647033] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:24.647293] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:24.647321] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): picking the request for winding >[2018-08-18 11:35:24.647347] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:24.647402] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:24.647403] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:24.647420] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:24.647458] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:24.647835] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:24.648025] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:24.648200] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:24.648232] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:24.648308] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:24.648516] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:24.648713] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:24.648894] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:24.648920] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): picking the request for winding >[2018-08-18 11:35:24.649267] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): destroying request, removing from all queues >[2018-08-18 11:35:24.649387] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:27.803286] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:27.803331] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:27.803381] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t at vol0-replicate-24 >[2018-08-18 11:35:27.803774] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:27.803852] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t returned with op_ret 0 >[2018-08-18 11:35:27.804447] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7fc073b634e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7fc082b88631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7fc07374e820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:27.804483] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:27.804518] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: 6135251d-bd90-46da-a007-650da6fbe9d0: generation now vs cached: 3, 3 >[2018-08-18 11:35:27.804757] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:27.805383] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:27.805413] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:27.805432] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:27.805661] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:27.806399] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:27.806622] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:27.806832] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:27.808954] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:27.808992] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:27.809013] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:27.809081] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:27.809100] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:27.810421] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:27.810462] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:27.810483] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:27.810941] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:27.811137] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 11:35:27.811164] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:27.811334] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:27.811692] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:27.811720] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): picking the request for winding >[2018-08-18 11:35:27.811741] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:27.811754] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:27.811810] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:27.811826] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:27.812042] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:27.812250] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:27.812569] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:27.812580] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:27.812596] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): picking the request for winding >[2018-08-18 11:35:27.812635] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:27.812649] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:27.812670] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:27.812734] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:27.812752] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:27.812968] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:27.813155] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:27.813467] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:27.813477] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:27.813506] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): picking the request for winding >[2018-08-18 11:35:27.813528] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:27.813582] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 11:35:27.813559] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:27.813581] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:27.813640] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:27.813818] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:27.813987] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:27.814136] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:27.814163] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:27.814380] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:27.814539] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:27.814703] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:27.814727] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): picking the request for winding >[2018-08-18 11:35:27.815044] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): destroying request, removing from all queues >[2018-08-18 11:35:27.815154] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:30.595523] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:30.595568] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:30.596045] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:30.596096] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 11:35:30.595672] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t at vol0-replicate-24 >[2018-08-18 11:35:30.596094] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t returned with op_ret 0 >[2018-08-18 11:35:30.596731] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7fc073b634e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7fc082b88631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7fc07374e820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:30.596768] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:30.596807] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: 6135251d-bd90-46da-a007-650da6fbe9d0: generation now vs cached: 3, 3 >[2018-08-18 11:35:30.597062] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:30.597781] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:30.597821] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:30.597845] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:30.598082] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:30.598691] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:30.598811] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:30.598923] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:30.600509] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:30.600546] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:30.600569] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:30.600649] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:30.600673] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:30.601842] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:30.601882] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:30.601905] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:30.602326] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:30.602516] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 11:35:30.602549] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:30.602742] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:30.603039] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:30.603251] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:30.603565] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:30.603606] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): picking the request for winding >[2018-08-18 11:35:30.603618] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:30.603633] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): picking the request for winding >[2018-08-18 11:35:30.603650] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:30.603710] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:30.603734] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:30.603864] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:30.604217] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:30.604415] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:30.604685] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] 0-vol0-write-behind: processing queues >[2018-08-18 11:35:30.604718] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:30.604739] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:30.604754] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:30.604784] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:30.604821] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:30.605039] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:30.605220] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:30.605460] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:30.605486] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): picking the request for winding >[2018-08-18 11:35:30.605510] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:30.605565] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:30.605562] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:30.605593] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:30.605607] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:30.606091] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:30.606304] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:30.606497] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:30.606526] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): picking the request for winding >[2018-08-18 11:35:30.606876] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): destroying request, removing from all queues >[2018-08-18 11:35:30.607012] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:32.621126] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /SR/Pn/uao3/mofunsky-img-sh1a_7 on vol0-replicate-31 >[2018-08-18 11:35:32.621612] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /SR/Pn/uao3/mofunsky-img-sh1a_7 with op_ret 0 >[2018-08-18 11:35:32.626206] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /SR/Pn/uao3/mofunsky-img-sh1a_7/5bdd176aa27211e897a0246e96313454.t on vol0-replicate-33 >[2018-08-18 11:35:32.626553] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /SR/Pn/uao3/mofunsky-img-sh1a_7/5bdd176aa27211e897a0246e96313454.t with op_ret 0 >[2018-08-18 11:35:32.626588] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-33 >[2018-08-18 11:35:32.626929] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-33: 886b3354-3d3b-47a8-9c92-0695974475c9: generation now vs cached: 3, 3 >[2018-08-18 11:35:32.627753] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:32.627787] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=0): picking the request for winding >[2018-08-18 11:35:32.627806] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:32.628038] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-33: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:32.628656] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:32.628808] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:32.629022] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:32.629071] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=1): picking the request for winding >[2018-08-18 11:35:32.629096] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:32.629161] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-33: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:32.629180] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 11:35:32.629221] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:32.629860] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:32.630050] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:32.630280] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:32.630308] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=0): picking the request for winding >[2018-08-18 11:35:32.630328] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:32.630394] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-33: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:32.630414] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 11:35:32.630439] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:32.630473] D [socket.c:532:__socket_rwv] 0-vol0-client-99: would have passed zero length to read/write >[2018-08-18 11:35:32.630512] D [socket.c:532:__socket_rwv] 0-vol0-client-100: would have passed zero length to read/write >[2018-08-18 11:35:32.630538] D [socket.c:532:__socket_rwv] 0-vol0-client-101: would have passed zero length to read/write >[2018-08-18 11:35:32.630905] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:32.630972] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:32.631126] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:32.631190] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=1): picking the request for winding >[2018-08-18 11:35:32.631216] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:32.631284] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-33: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:32.631304] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 11:35:32.631304] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:32.631883] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:32.631896] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:32.632072] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:32.632111] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=2): picking the request for winding >[2018-08-18 11:35:32.632132] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:32.632189] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-33: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:32.632217] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 11:35:32.632231] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:32.632715] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:32.632728] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:32.632893] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:32.632932] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=3): picking the request for winding >[2018-08-18 11:35:32.632953] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:32.633007] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-33: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:32.633026] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 11:35:32.633048] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:32.633606] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:32.633613] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:32.633794] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:32.633833] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=4): picking the request for winding >[2018-08-18 11:35:32.633958] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:32.634175] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=886b3354-3d3b-47a8-9c92-0695974475c9, gen=4): destroying request, removing from all queues >[2018-08-18 11:35:32.634312] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 11:35:33.584954] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:33.585000] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:33.585050] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t at vol0-replicate-24 >[2018-08-18 11:35:33.585428] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:33.585511] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t returned with op_ret 0 >[2018-08-18 11:35:33.586139] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7fc073b634e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7fc082b88631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7fc07374e820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:33.586171] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:33.586206] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: 6135251d-bd90-46da-a007-650da6fbe9d0: generation now vs cached: 3, 3 >[2018-08-18 11:35:33.586438] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:33.587199] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:33.587262] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:33.587298] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:33.587681] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:33.588659] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:33.589023] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:33.589331] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:33.590614] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:33.590656] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:33.590680] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:33.590753] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:33.590774] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:33.592032] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:33.592068] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:33.592088] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:33.592507] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:33.592734] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 11:35:33.592767] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:33.592962] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:33.593345] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:33.593543] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:33.593886] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:33.593901] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:33.593940] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): picking the request for winding >[2018-08-18 11:35:33.593962] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): picking the request for winding >[2018-08-18 11:35:33.593978] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:33.594046] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:33.594065] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:33.594204] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:33.594566] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:33.594760] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:33.594996] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:33.595021] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:33.595054] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:33.595115] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:33.595151] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:33.595152] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:33.595360] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:33.595510] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:33.595739] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:33.595764] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): picking the request for winding >[2018-08-18 11:35:33.595785] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:33.595842] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:33.595858] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:33.595857] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:33.595896] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:33.596305] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:33.596486] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:33.596668] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:33.596695] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): picking the request for winding >[2018-08-18 11:35:33.597047] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): destroying request, removing from all queues >[2018-08-18 11:35:33.597163] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:36.441952] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:36.442034] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 11:35:36.441997] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:36.442033] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t at vol0-replicate-24 >[2018-08-18 11:35:36.442461] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:36.442516] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t returned with op_ret 0 >[2018-08-18 11:35:36.443146] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7fc073b634e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7fc082b88631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7fc07374e820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:36.443182] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:36.443217] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: 6135251d-bd90-46da-a007-650da6fbe9d0: generation now vs cached: 3, 3 >[2018-08-18 11:35:36.443477] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:36.444103] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:36.444137] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:36.444157] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:36.444404] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:36.445061] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:36.445252] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:36.445425] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:36.446907] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:36.446941] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:36.446961] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:36.447017] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:36.448349] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:36.448396] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 11:35:36.447036] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:36.448395] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:36.448438] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:36.448879] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:36.449103] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 11:35:36.449135] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:36.449351] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:36.449518] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:36.449804] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:36.449818] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:36.449846] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): picking the request for winding >[2018-08-18 11:35:36.449899] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 11:35:36.449893] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:36.449899] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:36.450202] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:36.450395] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:36.450559] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:36.450609] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): picking the request for winding >[2018-08-18 11:35:36.450637] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:36.450712] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:36.450734] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:36.450736] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:36.451262] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:36.451276] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:36.451494] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:36.451535] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): picking the request for winding >[2018-08-18 11:35:36.451557] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:36.451625] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:36.451646] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:36.451683] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:36.452253] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:36.452265] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:36.452422] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:36.452465] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): picking the request for winding >[2018-08-18 11:35:36.452579] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:36.452810] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:36.452826] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): destroying request, removing from all queues >[2018-08-18 11:35:38.935212] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:38.935276] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:38.935336] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t at vol0-replicate-24 >[2018-08-18 11:35:38.935715] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:38.935766] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t returned with op_ret 0 >[2018-08-18 11:35:38.936416] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7fc073b634e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7fc082b88631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7fc07374e820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:38.936456] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:38.936494] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: 6135251d-bd90-46da-a007-650da6fbe9d0: generation now vs cached: 3, 3 >[2018-08-18 11:35:38.936764] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:38.937522] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:38.937561] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:38.937585] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:38.937889] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:38.938688] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:38.938900] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:38.939283] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:38.939301] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:38.939339] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:38.939363] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:38.939429] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:38.939449] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:38.940360] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:38.940394] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:38.940414] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:38.940727] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:38.940921] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:38.941211] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:38.941325] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:38.941442] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 11:35:38.941472] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): picking the request for winding >[2018-08-18 11:35:38.941687] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:38.941762] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:38.941783] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:38.941984] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:38.942016] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:38.942038] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:38.942274] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:38.942465] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:38.942742] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] 0-vol0-write-behind: processing queues >[2018-08-18 11:35:38.942773] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): picking the request for winding >[2018-08-18 11:35:38.942795] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): picking the request for winding >[2018-08-18 11:35:38.942810] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:38.942816] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:38.942874] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:38.942894] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:38.943031] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:38.943228] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:38.943417] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:38.943612] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:38.943672] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:38.943811] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:38.943912] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 11:35:38.943847] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:38.943911] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:38.944147] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:38.944323] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:38.944496] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:38.944522] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): picking the request for winding >[2018-08-18 11:35:38.944895] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): destroying request, removing from all queues >[2018-08-18 11:35:38.945034] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:42.026635] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.026664] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:42.026697] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t at vol0-replicate-24 >[2018-08-18 11:35:42.026923] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:42.027133] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t returned with op_ret 0 >[2018-08-18 11:35:42.027543] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7fc073b634e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7fc082b88631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7fc07374e820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.027560] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:42.027582] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: 6135251d-bd90-46da-a007-650da6fbe9d0: generation now vs cached: 3, 3 >[2018-08-18 11:35:42.027719] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:42.028447] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.028485] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:42.028510] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:42.028810] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:42.029542] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:42.029761] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:42.029982] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.030011] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:42.030031] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:42.030073] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.030103] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:42.030125] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:42.030581] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.030618] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:42.030634] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:42.030998] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.031149] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 11:35:42.031171] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:42.031311] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:42.031607] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.031640] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 11:35:42.031631] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): picking the request for winding >[2018-08-18 11:35:42.031639] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:42.031672] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:42.031718] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:42.032011] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.032035] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:42.032047] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:42.032058] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:42.032350] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.032373] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:42.032386] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): picking the request for winding >[2018-08-18 11:35:42.032397] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:42.032437] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:42.032450] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:42.032674] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:42.032885] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:42.033078] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.033105] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): picking the request for winding >[2018-08-18 11:35:42.033128] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:42.033175] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:42.033503] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:42.033685] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:42.033850] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.033882] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:42.033947] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:42.034136] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:42.034293] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:42.034444] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.034470] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:42.034530] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:42.034722] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:42.034884] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:42.035037] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.035060] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): picking the request for winding >[2018-08-18 11:35:42.035364] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): destroying request, removing from all queues >[2018-08-18 11:35:42.035475] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:42.094086] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /SR/Pn/uao3/mofunsky-img-sh1a_7/fb6eaeb8a22a11e8bffe246e96318264.t on vol0-replicate-7 >[2018-08-18 11:35:42.094566] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /SR/Pn/uao3/mofunsky-img-sh1a_7/fb6eaeb8a22a11e8bffe246e96318264.t with op_ret 0 >[2018-08-18 11:35:42.094612] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-7 >[2018-08-18 11:35:42.094862] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-7: 8bddf33d-7a82-45f3-8c98-9c08c1f773a3: generation now vs cached: 3, 3 >[2018-08-18 11:35:42.095710] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.095750] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=0): picking the request for winding >[2018-08-18 11:35:42.095772] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:42.096062] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:42.096690] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.096817] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:42.096978] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:42.097019] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=1): picking the request for winding >[2018-08-18 11:35:42.097046] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:42.097116] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:42.097137] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 11:35:42.097191] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.098053] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.098065] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:42.098247] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:42.098286] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=2): picking the request for winding >[2018-08-18 11:35:42.098311] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:42.098386] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:42.098401] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.098406] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 11:35:42.098496] D [socket.c:532:__socket_rwv] 0-vol0-client-21: would have passed zero length to read/write >[2018-08-18 11:35:42.098528] D [socket.c:532:__socket_rwv] 0-vol0-client-22: would have passed zero length to read/write >[2018-08-18 11:35:42.098553] D [socket.c:532:__socket_rwv] 0-vol0-client-23: would have passed zero length to read/write >[2018-08-18 11:35:42.098913] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:42.099123] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:42.099461] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.099476] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.099513] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=0): picking the request for winding >[2018-08-18 11:35:42.099533] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:42.099594] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:42.099623] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 11:35:42.100155] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.100191] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:42.100389] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:42.100432] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=1): picking the request for winding >[2018-08-18 11:35:42.100452] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:42.100503] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:42.100525] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 11:35:42.100571] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.100944] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.101048] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:42.101258] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:42.101304] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=2): picking the request for winding >[2018-08-18 11:35:42.101323] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:42.101374] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:42.101390] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 11:35:42.101447] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.101859] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:42.101971] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.102069] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:42.102113] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=3): picking the request for winding >[2018-08-18 11:35:42.102256] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:42.102432] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:42.102569] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 11:35:45.091517] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:45.091565] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:45.091632] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t at vol0-replicate-24 >[2018-08-18 11:35:45.092030] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:45.092117] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t returned with op_ret 0 >[2018-08-18 11:35:45.092783] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7fc073b634e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7fc082b88631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7fc07374e820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:45.092819] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:45.092858] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: 6135251d-bd90-46da-a007-650da6fbe9d0: generation now vs cached: 3, 3 >[2018-08-18 11:35:45.093113] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:45.093830] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:45.093861] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:45.093879] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:45.094106] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:45.094845] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:45.095037] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:45.095199] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:45.097452] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:45.097488] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:45.097510] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:45.097572] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:45.097592] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:45.098915] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:45.098951] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:45.098971] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:45.099384] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:45.099588] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 11:35:45.099630] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:45.099819] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:45.100200] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:45.100295] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:45.100453] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:45.100493] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): picking the request for winding >[2018-08-18 11:35:45.100514] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): picking the request for winding >[2018-08-18 11:35:45.100529] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:45.100605] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:45.100615] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:45.100643] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:45.100788] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:45.101159] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:45.101353] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:45.101695] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:45.101729] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:45.101734] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:45.101747] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:45.101789] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:45.101878] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:45.102161] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:45.102393] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:45.102753] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:45.102783] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:45.102793] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): picking the request for winding >[2018-08-18 11:35:45.102821] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:45.102875] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 11:35:45.102853] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:45.102874] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:45.102907] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:45.103304] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:45.103450] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:45.103580] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:45.103612] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): picking the request for winding >[2018-08-18 11:35:45.103888] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): destroying request, removing from all queues >[2018-08-18 11:35:45.103984] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:48.170836] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t on vol0-replicate-24 >[2018-08-18 11:35:48.171346] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.17:49212: ping timer event already removed >[2018-08-18 11:35:48.171647] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.27:49212: ping timer event already removed >[2018-08-18 11:35:48.171921] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fc08300bd0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7fc08300c331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7fc0830090f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7fc0783d3342] ))))) 0-: 10.16.71.37:49212: ping timer event already removed >[2018-08-18 11:35:48.172029] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t with op_ret 0 >[2018-08-18 11:35:48.172062] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-24 >[2018-08-18 11:35:48.172377] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: 6135251d-bd90-46da-a007-650da6fbe9d0: generation now vs cached: 3, 3 >[2018-08-18 11:35:48.173248] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:48.173286] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:48.173305] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:48.173571] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:48.174293] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:48.174481] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:48.174691] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:48.176437] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:48.176476] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:48.176496] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:48.176561] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:48.176578] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:48.177679] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:48.177701] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:48.177712] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:48.177945] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:48.178055] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 11:35:48.178071] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:48.178171] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:48.178388] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:48.178403] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): picking the request for winding >[2018-08-18 11:35:48.178416] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:48.178424] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:48.178459] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:48.178680] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:48.178700] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:48.178710] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:48.178718] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:48.178935] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:48.178952] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:48.178965] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:48.178974] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:48.179491] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:48.179714] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:48.179898] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:48.179934] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): picking the request for winding >[2018-08-18 11:35:48.179953] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): picking the request for winding >[2018-08-18 11:35:48.179972] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:48.180019] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:48.180036] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:48.180160] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:48.180504] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:48.180696] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:48.180863] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:48.180890] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:48.180953] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:48.181141] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:48.181299] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:48.181450] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:48.181482] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:48.181539] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:48.181735] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:48.181885] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:48.182034] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:48.182057] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): picking the request for winding >[2018-08-18 11:35:48.182360] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): destroying request, removing from all queues >[2018-08-18 11:35:48.182472] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:51.049952] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:51.049997] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:51.050051] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t at vol0-replicate-24 >[2018-08-18 11:35:51.050412] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:51.050494] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t returned with op_ret 0 >[2018-08-18 11:35:51.051095] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7fc073b634e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7fc082b88631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7fc07374e820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:51.051132] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:51.051168] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: 6135251d-bd90-46da-a007-650da6fbe9d0: generation now vs cached: 3, 3 >[2018-08-18 11:35:51.051469] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:51.052186] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:51.052223] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:51.052244] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:51.052506] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:51.053266] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:51.053499] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:51.053723] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:51.055066] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:51.055102] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:51.055124] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:51.055184] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:51.055205] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:51.056296] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:51.056331] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:51.056351] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:51.056771] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:51.056985] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 11:35:51.057015] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:51.057201] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:51.057452] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:51.057651] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:51.057901] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:51.057929] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): picking the request for winding >[2018-08-18 11:35:51.057952] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): picking the request for winding >[2018-08-18 11:35:51.057974] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:51.057990] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:51.058043] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:51.058062] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:51.058192] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:51.058308] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:51.058338] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:51.058359] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:51.058586] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:51.058770] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:51.059140] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:51.059155] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:51.059190] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): picking the request for winding >[2018-08-18 11:35:51.059217] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:51.059274] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 11:35:51.059250] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=2, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:51.059273] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:51.059312] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:51.059549] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:51.059737] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:51.059902] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:51.059932] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 11:35:51.059313] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:51.059931] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=6135251d-bd90-46da-a007-650da6fbe9d0. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:51.060186] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:51.060349] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:51.060512] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:51.060536] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): picking the request for winding >[2018-08-18 11:35:51.060878] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): destroying request, removing from all queues >[2018-08-18 11:35:51.060987] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:53.799117] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:53.799165] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:53.799231] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t at vol0-replicate-24 >[2018-08-18 11:35:53.799658] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:53.799719] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t returned with op_ret 0 >[2018-08-18 11:35:53.800374] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7fc073b634e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7fc082b88631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7fc07374e820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:53.800417] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:53.800458] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: 6135251d-bd90-46da-a007-650da6fbe9d0: generation now vs cached: 3, 3 >[2018-08-18 11:35:53.800763] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:53.801512] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:53.801551] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:53.801573] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:53.801849] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:53.802703] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:53.802953] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:53.803159] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:53.804071] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:53.804110] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:53.804133] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:53.804209] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:53.804231] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:53.805459] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:53.805511] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 11:35:53.805503] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:53.805510] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:53.806072] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:53.806277] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:53.806475] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:53.806818] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:53.806836] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 11:35:53.806870] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): picking the request for winding >[2018-08-18 11:35:53.807076] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:53.807148] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:53.807675] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:53.807737] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:53.807894] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:53.807943] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 11:35:53.807168] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:53.807942] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): picking the request for winding >[2018-08-18 11:35:53.807982] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:53.808050] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 11:35:53.808043] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:53.808050] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:53.808083] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:53.808503] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:53.808590] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:53.808827] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:53.808878] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): picking the request for winding >[2018-08-18 11:35:53.808900] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:53.808955] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:53.808972] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:53.809033] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:53.809443] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:53.809660] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:53.810071] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:53.810089] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:53.810101] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:53.810472] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:53.810627] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:56.643767] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:56.643819] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:56.643883] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t at vol0-replicate-24 >[2018-08-18 11:35:56.644346] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:56.644436] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t returned with op_ret 0 >[2018-08-18 11:35:56.645174] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7fc073b634e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7fc082b88631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7fc07374e820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:56.645218] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:56.645259] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: 6135251d-bd90-46da-a007-650da6fbe9d0: generation now vs cached: 3, 3 >[2018-08-18 11:35:56.645546] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:56.646289] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:56.646327] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:56.646351] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:56.646632] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:56.647424] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:56.647643] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:56.647851] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:56.649343] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:56.649382] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:56.649405] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:56.649471] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:56.649491] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:56.651197] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:56.651252] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:56.651280] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:56.651829] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:56.651949] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:56.652173] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:56.652541] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:56.652562] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 11:35:56.652612] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): picking the request for winding >[2018-08-18 11:35:56.652855] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:56.652949] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:56.652974] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:56.653673] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:56.653694] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:56.653915] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:56.653970] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): picking the request for winding >[2018-08-18 11:35:56.654057] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 11:35:56.653996] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:56.654056] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:56.654105] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:56.654126] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:56.654726] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:56.654746] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:56.654964] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:56.655012] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): picking the request for winding >[2018-08-18 11:35:56.655045] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:56.655111] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:56.655154] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:56.655177] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:56.655894] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:56.655914] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:56.656128] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:56.656177] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): picking the request for winding >[2018-08-18 11:35:56.656319] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:56.656586] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:56.656594] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=4): destroying request, removing from all queues >[2018-08-18 11:35:57.644820] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:57.644864] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=0): picking the request for winding >[2018-08-18 11:35:57.644914] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/sjkzhdd_1/38d9b179a2d711e8b1a76c92bf6dcc7a.t at vol0-replicate-17 >[2018-08-18 11:35:57.645284] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:57.645361] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/sjkzhdd_1/38d9b179a2d711e8b1a76c92bf6dcc7a.t returned with op_ret 0 >[2018-08-18 11:35:57.645957] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7fc073b634e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7fc082b88631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7fc07374e820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:57.645989] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=0): picking the request for winding >[2018-08-18 11:35:57.646023] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-17: 80d5c203-5fc5-4a6a-bced-18f47e8ad625: generation now vs cached: 3, 3 >[2018-08-18 11:35:57.646255] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:57.646951] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:57.646991] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=0): picking the request for winding >[2018-08-18 11:35:57.647011] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:57.647279] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-17: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:57.647954] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:57.648145] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:57.648315] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:57.649186] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:57.649220] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=0): picking the request for winding >[2018-08-18 11:35:57.649240] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:57.649299] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-17: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:57.649318] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 11:35:57.650356] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:57.650383] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625): ordering.go is not set, hence not winding >[2018-08-18 11:35:57.650399] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:57.650707] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:57.650845] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 11:35:57.650862] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625): ordering.go is not set, hence not winding >[2018-08-18 11:35:57.650973] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7fc073b616a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7fc073b62a90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:57.651187] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:57.651351] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:57.651576] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:57.651596] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=1): picking the request for winding >[2018-08-18 11:35:57.651615] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=2): picking the request for winding >[2018-08-18 11:35:57.651623] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:57.651623] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:57.651665] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-17: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:57.651678] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 11:35:57.651780] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-17: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:57.652011] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:57.652091] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:57.652245] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:57.652283] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625): ordering.go is not set, hence not winding >[2018-08-18 11:35:57.652295] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:57.652409] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:57.652432] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=3): picking the request for winding >[2018-08-18 11:35:57.652453] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625. CONFLICT: unique=0, fop=WRITE, gen=2, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:57.652506] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-17: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:57.652522] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 11:35:57.652551] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7fc073b63938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7fc082b88161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:57.652572] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625. CONFLICT: unique=0, fop=WRITE, gen=2, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:57.652864] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:57.653018] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:57.653164] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:57.653191] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 11:35:57.653253] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 11:35:57.653428] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:57.653574] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:57.653730] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:57.653753] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=4): picking the request for winding >[2018-08-18 11:35:57.654037] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=80d5c203-5fc5-4a6a-bced-18f47e8ad625, gen=4): destroying request, removing from all queues >[2018-08-18 11:35:57.654144] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 11:35:59.629044] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7fc073b65880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7fc082b893bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7fc07374c596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:59.629091] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:59.629148] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t at vol0-replicate-24 >[2018-08-18 11:35:59.629585] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:59.629678] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/cpsbxt_5/653ed09ba2da11e880ef246e963180cc.t returned with op_ret 0 >[2018-08-18 11:35:59.630280] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7fc073b634e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7fc082b88631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7fc07374e820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:59.630315] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:59.630350] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: 6135251d-bd90-46da-a007-650da6fbe9d0: generation now vs cached: 3, 3 >[2018-08-18 11:35:59.630660] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:59.631326] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:59.631371] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:59.631395] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:59.631672] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:59.632587] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:59.632896] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:59.633173] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:59.634611] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:59.634646] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:59.634666] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 11:35:59.634727] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:59.634745] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:59.636217] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:59.636271] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:59.636297] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 11:35:59.636803] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:59.637015] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7fc073b620b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7fc073b62a5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 11:35:59.637045] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6135251d-bd90-46da-a007-650da6fbe9d0): ordering.go is not set, hence not winding >[2018-08-18 11:35:59.637312] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:59.637514] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:59.637794] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:59.637836] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:59.637867] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): picking the request for winding >[2018-08-18 11:35:59.637922] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:59.637941] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:59.638202] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:59.638382] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:59.638567] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=1): destroying request, removing from all queues >[2018-08-18 11:35:59.638613] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): picking the request for winding >[2018-08-18 11:35:59.638639] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 11:35:59.638712] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:59.638732] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:59.638764] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:59.639191] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:59.639284] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7fc073b63224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7fc082b891e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7fc07374d173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:59.639394] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=2): destroying request, removing from all queues >[2018-08-18 11:35:59.639436] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): picking the request for winding >[2018-08-18 11:35:59.639456] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 11:35:59.639517] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 11:35:59.639535] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 11:35:59.639575] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:59.639959] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7fc073b60827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 11:35:59.640135] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7fc073b60857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7fc073b60b88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7fc073b62b6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=3): destroying request, removing from all queues >[2018-08-18 11:35:59.640503] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:59.640518] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7fc073b62adb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7fc073b62b77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7fc073dce4cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7fc078167812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 11:35:59.640551] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): picking the request for winding >[2018-08-18 11:35:59.640902] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc082b0ffab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7fc073b5c582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7fc073b5e6d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7fc073b62991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7fc073b62a9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=6135251d-bd90-46da-a007-650da6fbe9d0, gen=0): destroying request, removing from all queues >[2018-08-18 11:35:59.640905] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction
<b>You cannot view the attachment while viewing its details because your browser does not support IFRAMEs. <a href="attachment.cgi?id=1476762">View the attachment on a separate page</a>.</b>
View Attachment As Raw
Actions:
View
Attachments on
bug 1618932
: 1476762 |
1476763
|
1476765
|
1477213
|
1485128
|
1485129
|
1487508