Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 1476753 Details for
Bug 1618925
dht_create: no subvolume in layout
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
gfapi log
gfapi_20180818_1805.log (text/plain), 2.54 MB, created by
frostyplanet
on 2018-08-18 11:35:09 UTC
(
hide
)
Description:
gfapi log
Filename:
MIME Type:
Creator:
frostyplanet
Created:
2018-08-18 11:35:09 UTC
Size:
2.54 MB
patch
obsolete
>[2018-08-18 10:05:00.448599] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.448644] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:00.448691] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /7A/Tp/bSUB/my-bucket_0/79477581a2c811e8bffe246e96318264.t at vol0-replicate-14 >[2018-08-18 10:05:00.449188] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:00.449252] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /7A/Tp/bSUB/my-bucket_0/79477581a2c811e8bffe246e96318264.t returned with op_ret 0 >[2018-08-18 10:05:00.449802] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7f3d0256d4e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7f3d0d44d631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7f3d02158820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.449836] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:00.449870] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-14: f3c71c0d-9173-488b-933b-386804975422: generation now vs cached: 3, 3 >[2018-08-18 10:05:00.450148] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:00.450776] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.450809] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:00.450830] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:00.451127] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:00.452005] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:00.452228] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:00.452433] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.454090] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.454128] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:00.454150] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:00.454229] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:00.454248] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:00.455495] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.455533] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=f3c71c0d-9173-488b-933b-386804975422): ordering.go is not set, hence not winding >[2018-08-18 10:05:00.455554] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 10:05:00.455967] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.456184] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7f3d0256c0b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7f3d0256ca5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 10:05:00.456219] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=f3c71c0d-9173-488b-933b-386804975422): ordering.go is not set, hence not winding >[2018-08-18 10:05:00.456405] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7f3d0256b6a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7f3d0256ca90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:00.456769] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.456942] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:00.457150] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:00.457188] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): picking the request for winding >[2018-08-18 10:05:00.457209] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): picking the request for winding >[2018-08-18 10:05:00.457222] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 10:05:00.457277] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:00.457302] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:00.457322] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.457424] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:00.457877] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.457943] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:00.458211] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:00.458288] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=f3c71c0d-9173-488b-933b-386804975422): ordering.go is not set, hence not winding >[2018-08-18 10:05:00.458321] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 10:05:00.458402] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.458430] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=f3c71c0d-9173-488b-933b-386804975422): ordering.go is not set, hence not winding >[2018-08-18 10:05:00.458502] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:00.458837] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.458871] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:00.459060] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:00.459103] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): picking the request for winding >[2018-08-18 10:05:00.459130] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=f3c71c0d-9173-488b-933b-386804975422. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=f3c71c0d-9173-488b-933b-386804975422, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:00.459179] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:00.459197] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:00.459235] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.459266] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=f3c71c0d-9173-488b-933b-386804975422. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=f3c71c0d-9173-488b-933b-386804975422, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:00.459611] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:00.459776] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): destroying request, removing from all queues >[2018-08-18 10:05:00.459935] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.459957] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=4): picking the request for winding >[2018-08-18 10:05:00.460278] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=4): destroying request, removing from all queues >[2018-08-18 10:05:00.460393] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:00.654980] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.655020] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): picking the request for winding >[2018-08-18 10:05:00.655088] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/sjkzhdd_1/6d48364fa2cb11e880ef246e963180cc.t at vol0-replicate-21 >[2018-08-18 10:05:00.655481] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:00.655600] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/sjkzhdd_1/6d48364fa2cb11e880ef246e963180cc.t returned with op_ret 0 >[2018-08-18 10:05:00.656089] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7f3d0256d4e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7f3d0d44d631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7f3d02158820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.656118] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): picking the request for winding >[2018-08-18 10:05:00.656147] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-21: ebb1b4d9-763b-4b2e-9654-888177709dc4: generation now vs cached: 3, 3 >[2018-08-18 10:05:00.656365] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:00.656958] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.656988] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): picking the request for winding >[2018-08-18 10:05:00.657006] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:00.657279] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:00.658116] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:00.658341] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:00.658555] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.659868] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.659900] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): picking the request for winding >[2018-08-18 10:05:00.659918] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:00.659965] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:00.659981] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:00.661052] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.661085] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4): ordering.go is not set, hence not winding >[2018-08-18 10:05:00.661101] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 10:05:00.661473] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.661673] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7f3d0256c0b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7f3d0256ca5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 10:05:00.661703] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4): ordering.go is not set, hence not winding >[2018-08-18 10:05:00.661881] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7f3d0256b6a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7f3d0256ca90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:00.662221] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.662248] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=1): picking the request for winding >[2018-08-18 10:05:00.662269] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=WRITE, gen=2, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:00.662281] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 10:05:00.662330] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:00.662672] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.662860] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:00.663067] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:00.663132] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=2): picking the request for winding >[2018-08-18 10:05:00.663152] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4): ordering.go is not set, hence not winding >[2018-08-18 10:05:00.663164] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 10:05:00.663271] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:00.663287] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:00.663287] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.663316] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4): ordering.go is not set, hence not winding >[2018-08-18 10:05:00.663629] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:00.663813] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:00.664002] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.664027] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=3): picking the request for winding >[2018-08-18 10:05:00.664070] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4. CONFLICT: unique=0, fop=WRITE, gen=2, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:00.664125] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:00.664146] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:00.664169] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.664205] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4. CONFLICT: unique=0, fop=WRITE, gen=2, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:00.664515] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:00.664698] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:00.664872] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.664901] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:00.664970] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:00.665188] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:00.665373] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=3): destroying request, removing from all queues >[2018-08-18 10:05:00.665544] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:00.665568] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=4): picking the request for winding >[2018-08-18 10:05:00.665919] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=4): destroying request, removing from all queues >[2018-08-18 10:05:00.666047] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:03.103801] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS on vol0-replicate-24 >[2018-08-18 10:05:03.104026] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS on vol0-replicate-24 >[2018-08-18 10:05:03.104294] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS with op_ret 0 >[2018-08-18 10:05:03.104876] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS with op_ret 0 >[2018-08-18 10:05:03.110878] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt on vol0-replicate-34 >[2018-08-18 10:05:03.111303] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt with op_ret 0 >[2018-08-18 10:05:03.112205] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt on vol0-replicate-34 >[2018-08-18 10:05:03.112562] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt with op_ret 0 >[2018-08-18 10:05:03.115443] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx on vol0-replicate-20 >[2018-08-18 10:05:03.115828] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx with op_ret 0 >[2018-08-18 10:05:03.117183] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx on vol0-replicate-20 >[2018-08-18 10:05:03.117555] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx with op_ret 0 >[2018-08-18 10:05:03.120054] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0 on vol0-replicate-23 >[2018-08-18 10:05:03.120519] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0 with op_ret 0 >[2018-08-18 10:05:03.121720] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0 on vol0-replicate-23 >[2018-08-18 10:05:03.122142] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0 with op_ret 0 >[2018-08-18 10:05:03.125277] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m on vol0-replicate-30 >[2018-08-18 10:05:03.125786] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m with op_ret 0 >[2018-08-18 10:05:03.126300] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:03.126804] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:03.126847] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m on vol0-replicate-30 >[2018-08-18 10:05:03.126863] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c001f40, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:03.127229] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m with op_ret 0 >[2018-08-18 10:05:03.127296] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c30001f40, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:03.127309] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-6: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:03.127333] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2c001f40, vol0-client-6 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:03.127364] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-7: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:03.127379] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2c001f40, vol0-client-7 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:03.127526] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-8: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:03.127566] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2c001f40, vol0-client-8 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:03.127870] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-18: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:03.127919] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c30001f40, vol0-client-18 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:03.127911] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:03.127925] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-20: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:03.127983] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c30001f40, vol0-client-20 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:03.128066] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-19: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:03.128101] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c30001f40, vol0-client-19 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:03.128284] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:03.128291] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:03.128464] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:03.128597] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34001f40, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:03.128927] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3000bcb0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:03.128969] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3000bcb0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:03.129111] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c28001f40, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:03.129490] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-6: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:03.129526] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c34001f40, vol0-client-6 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:03.129538] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-7: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:03.129548] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-19: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:03.129558] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-20: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:03.129581] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-18: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:03.129583] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c34001f40, vol0-client-7 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:03.129622] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c28001f40, vol0-client-18 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:03.129604] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c28001f40, vol0-client-19 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:03.129603] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c28001f40, vol0-client-20 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:03.129895] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-51: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:03.129923] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c3000bcb0, vol0-client-51 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:03.129950] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-53: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:03.129925] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-52: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:03.129984] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c3000bcb0, vol0-client-53 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:03.129997] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c3000bcb0, vol0-client-52 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:03.130007] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:03.130070] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:03.130567] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2800aea0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:03.130626] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2800aea0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:03.130891] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-51: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:03.130904] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-52: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:03.130919] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2800aea0, vol0-client-51 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:03.130934] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2800aea0, vol0-client-52 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:03.130945] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-53: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:03.130979] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2800aea0, vol0-client-53 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.028296] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-vol0-dht: Found anomalies in /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m (gfid = 417cdc92-cdda-4785-a8bb-98a38a36d6cd). Holes=2 overlaps=0 >[2018-08-18 10:05:04.028329] D [MSGID: 0] [dht-common.c:943:dht_lookup_dir_cbk] 0-vol0-dht: fixing assignment on /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m >[2018-08-18 10:05:04.028399] D [MSGID: 0] [dht-selfheal.c:1879:dht_selfheal_layout_new_directory] 0-vol0-dht: chunk size = 0xffffffff / 274638528 = 15.638619 >[2018-08-18 10:05:04.028420] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-19 >[2018-08-18 10:05:04.028430] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-2 >[2018-08-18 10:05:04.028441] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-20 >[2018-08-18 10:05:04.028450] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-21 >[2018-08-18 10:05:04.028460] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-22 >[2018-08-18 10:05:04.028469] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-23 >[2018-08-18 10:05:04.028478] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-24 >[2018-08-18 10:05:04.028487] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-25 >[2018-08-18 10:05:04.028496] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-26 >[2018-08-18 10:05:04.028506] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-27 >[2018-08-18 10:05:04.028515] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-28 >[2018-08-18 10:05:04.028524] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-29 >[2018-08-18 10:05:04.028533] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-3 >[2018-08-18 10:05:04.028542] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-30 >[2018-08-18 10:05:04.028551] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-31 >[2018-08-18 10:05:04.028560] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-32 >[2018-08-18 10:05:04.028570] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-33 >[2018-08-18 10:05:04.028579] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-34 >[2018-08-18 10:05:04.028588] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-35 >[2018-08-18 10:05:04.028597] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-4 >[2018-08-18 10:05:04.028606] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-5 >[2018-08-18 10:05:04.028622] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-6 >[2018-08-18 10:05:04.028632] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-7 >[2018-08-18 10:05:04.028641] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-8 >[2018-08-18 10:05:04.028650] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-9 >[2018-08-18 10:05:04.028660] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-0 >[2018-08-18 10:05:04.028669] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-1 >[2018-08-18 10:05:04.028678] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-10 >[2018-08-18 10:05:04.028687] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-11 >[2018-08-18 10:05:04.028696] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-12 >[2018-08-18 10:05:04.028705] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-13 >[2018-08-18 10:05:04.028713] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-14 >[2018-08-18 10:05:04.028722] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-15 >[2018-08-18 10:05:04.028731] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-16 >[2018-08-18 10:05:04.028740] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-17 >[2018-08-18 10:05:04.028750] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-18 >[2018-08-18 10:05:04.028993] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-0: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.029059] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-1: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.029202] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.029228] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.029291] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-12: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.029329] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 10:05:04.029325] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 10:05:04.029351] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-39: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.029385] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30022330, vol0-client-39 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.029386] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-40: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.029396] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-41: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.029445] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30022330, vol0-client-40 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.029459] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30022330, vol0-client-41 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.029473] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 10:05:04.029490] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 10:05:04.029476] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.029512] D [MSGID: 0] [afr-transaction.c:1877:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks failed. Proceeding to blocking >[2018-08-18 10:05:04.029522] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-45: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.029523] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 10:05:04.029563] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c3002ddf0, vol0-client-45 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.029603] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 10:05:04.029604] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-46: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.029631] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-47: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.029658] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c3002ddf0, vol0-client-47 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.029660] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c3002ddf0, vol0-client-46 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.029660] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-49: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.029696] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 10:05:04.029715] D [MSGID: 0] [afr-transaction.c:1877:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-15: Non blocking inodelks failed. Proceeding to blocking >[2018-08-18 10:05:04.029717] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-51: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.029706] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-48: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.029697] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-50: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.029748] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30039d10, vol0-client-51 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.029707] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30033fb0, vol0-client-49 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.029791] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30033fb0, vol0-client-48 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.029810] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30033fb0, vol0-client-50 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.029818] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-52: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.029846] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:04.029850] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:04.029847] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-53: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.029881] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30039d10, vol0-client-53 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.029864] D [MSGID: 0] [afr-transaction.c:1877:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks failed. Proceeding to blocking >[2018-08-18 10:05:04.029887] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30039d10, vol0-client-52 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.029893] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-55: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.029916] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 10:05:04.029898] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-54: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.029928] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30013410, vol0-client-55 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.029934] D [MSGID: 0] [afr-transaction.c:1877:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-17: Non blocking inodelks failed. Proceeding to blocking >[2018-08-18 10:05:04.029941] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30013410, vol0-client-54 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.029943] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-56: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.029960] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-57: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.029977] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-59: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.029970] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-58: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.029968] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30013410, vol0-client-56 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.029988] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c3000bcb0, vol0-client-57 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.030011] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c3000bcb0, vol0-client-58 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.030014] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 10:05:04.030011] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c3000bcb0, vol0-client-59 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.030063] D [MSGID: 0] [afr-transaction.c:1877:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-18: Non blocking inodelks failed. Proceeding to blocking >[2018-08-18 10:05:04.030061] D [MSGID: 0] [afr-lk-common.c:1189:afr_lock_blocking] 0-vol0-replicate-13: we're done locking >[2018-08-18 10:05:04.030064] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-7: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.030084] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 10:05:04.030055] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-6: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.030106] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30001f40, vol0-client-7 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.030103] D [MSGID: 0] [afr-transaction.c:1857:afr_post_blocking_inodelk_cbk] 0-vol0-replicate-13: Blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.030103] D [MSGID: 0] [afr-transaction.c:1877:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-19: Non blocking inodelks failed. Proceeding to blocking >[2018-08-18 10:05:04.030145] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30001f40, vol0-client-6 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.030147] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-8: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.030157] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-60: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.030175] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30001f40, vol0-client-8 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.030183] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c3000aea0, vol0-client-60 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.030187] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 10:05:04.030187] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-62: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.030188] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-64: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.030212] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-65: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.030216] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c3000aea0, vol0-client-62 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.030213] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-63: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.030230] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30009700, vol0-client-65 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.030183] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-61: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.030224] D [MSGID: 0] [afr-transaction.c:1877:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-2: Non blocking inodelks failed. Proceeding to blocking >[2018-08-18 10:05:04.030222] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30009700, vol0-client-64 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.030279] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30009700, vol0-client-63 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.030269] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c3000aea0, vol0-client-61 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.030292] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-66: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.030269] D [MSGID: 0] [afr-lk-common.c:1189:afr_lock_blocking] 0-vol0-replicate-15: we're done locking >[2018-08-18 10:05:04.030330] D [MSGID: 0] [afr-transaction.c:1857:afr_post_blocking_inodelk_cbk] 0-vol0-replicate-15: Blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.030342] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-68: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.030330] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-vol0-dht: Found anomalies in /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m (gfid = 417cdc92-cdda-4785-a8bb-98a38a36d6cd). Holes=2 overlaps=0 >[2018-08-18 10:05:04.030351] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-20: lk op is for a transaction >[2018-08-18 10:05:04.030355] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30044470, vol0-client-66 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.030362] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30044470, vol0-client-68 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.030295] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:04.030367] D [MSGID: 0] [dht-common.c:943:dht_lookup_dir_cbk] 0-vol0-dht: fixing assignment on /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m >[2018-08-18 10:05:04.030379] D [MSGID: 0] [afr-transaction.c:1877:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks failed. Proceeding to blocking >[2018-08-18 10:05:04.030384] D [MSGID: 0] [afr-transaction.c:1877:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-20: Non blocking inodelks failed. Proceeding to blocking >[2018-08-18 10:05:04.030404] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-67: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.030414] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 10:05:04.030429] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-72: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.030429] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30044470, vol0-client-67 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.030445] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-73: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.030469] D [MSGID: 0] [dht-selfheal.c:1879:dht_selfheal_layout_new_directory] 0-vol0-dht: chunk size = 0xffffffff / 274638528 = 15.638619 >[2018-08-18 10:05:04.030444] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30046700, vol0-client-72 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.030500] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-19 >[2018-08-18 10:05:04.030511] D [MSGID: 0] [afr-lk-common.c:1189:afr_lock_blocking] 0-vol0-replicate-16: we're done locking >[2018-08-18 10:05:04.030445] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-23: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.030518] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-2 >[2018-08-18 10:05:04.030523] D [MSGID: 0] [afr-transaction.c:1857:afr_post_blocking_inodelk_cbk] 0-vol0-replicate-16: Blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.030522] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30046700, vol0-client-73 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.030544] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-20 >[2018-08-18 10:05:04.030514] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-22: lk op is for a transaction >[2018-08-18 10:05:04.030562] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-21 >[2018-08-18 10:05:04.030547] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-74: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.030576] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-22 >[2018-08-18 10:05:04.030577] D [MSGID: 0] [afr-transaction.c:1877:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-22: Non blocking inodelks failed. Proceeding to blocking >[2018-08-18 10:05:04.030585] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c30046700, vol0-client-74 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.030592] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-23 >[2018-08-18 10:05:04.030589] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-25: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.030608] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-24 >[2018-08-18 10:05:04.030609] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 10:05:04.030624] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-25 >[2018-08-18 10:05:04.030628] D [MSGID: 0] [afr-transaction.c:1877:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks failed. Proceeding to blocking >[2018-08-18 10:05:04.030642] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-26 >[2018-08-18 10:05:04.030644] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 10:05:04.030658] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-27 >[2018-08-18 10:05:04.030675] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-28 >[2018-08-18 10:05:04.030690] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-29 >[2018-08-18 10:05:04.030679] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-79: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.030700] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-3 >[2018-08-18 10:05:04.030692] D [MSGID: 0] [afr-lk-common.c:1189:afr_lock_blocking] 0-vol0-replicate-17: we're done locking >[2018-08-18 10:05:04.030705] D [MSGID: 0] [afr-lk-common.c:1189:afr_lock_blocking] 0-vol0-replicate-19: we're done locking >[2018-08-18 10:05:04.030711] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-30 >[2018-08-18 10:05:04.030721] D [MSGID: 0] [afr-transaction.c:1857:afr_post_blocking_inodelk_cbk] 0-vol0-replicate-19: Blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.030724] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-31 >[2018-08-18 10:05:04.030725] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c300521c0, vol0-client-79 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.030740] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-32 >[2018-08-18 10:05:04.030740] D [MSGID: 0] [afr-lk-common.c:1189:afr_lock_blocking] 0-vol0-replicate-18: we're done locking >[2018-08-18 10:05:04.030732] D [MSGID: 0] [afr-transaction.c:1857:afr_post_blocking_inodelk_cbk] 0-vol0-replicate-17: Blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.030728] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-78: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.030755] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-33 >[2018-08-18 10:05:04.030725] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-80: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.030773] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-34 >[2018-08-18 10:05:04.030771] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c300521c0, vol0-client-78 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.030781] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c300521c0, vol0-client-80 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.030788] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-35 >[2018-08-18 10:05:04.030780] D [MSGID: 0] [afr-transaction.c:1857:afr_post_blocking_inodelk_cbk] 0-vol0-replicate-18: Blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.030796] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-26: lk op is for a transaction >[2018-08-18 10:05:04.030804] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-4 >[2018-08-18 10:05:04.030814] D [MSGID: 0] [afr-transaction.c:1877:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-26: Non blocking inodelks failed. Proceeding to blocking >[2018-08-18 10:05:04.030820] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-5 >[2018-08-18 10:05:04.030836] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-6 >[2018-08-18 10:05:04.030846] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-7 >[2018-08-18 10:05:04.030856] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-8 >[2018-08-18 10:05:04.030863] D [MSGID: 0] [afr-lk-common.c:1189:afr_lock_blocking] 0-vol0-replicate-2: we're done locking >[2018-08-18 10:05:04.030880] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-9 >[2018-08-18 10:05:04.030883] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-27: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.030893] D [MSGID: 0] [afr-transaction.c:1857:afr_post_blocking_inodelk_cbk] 0-vol0-replicate-2: Blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.030896] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-0 >[2018-08-18 10:05:04.030912] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-1 >[2018-08-18 10:05:04.030907] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:04.030923] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-23: lk op is for a transaction >[2018-08-18 10:05:04.030927] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-10 >[2018-08-18 10:05:04.030938] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-11 >[2018-08-18 10:05:04.030947] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-12 >[2018-08-18 10:05:04.030936] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-28: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.030956] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-13 >[2018-08-18 10:05:04.030937] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 10:05:04.030966] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-14 >[2018-08-18 10:05:04.030971] D [MSGID: 0] [afr-lk-common.c:1189:afr_lock_blocking] 0-vol0-replicate-21: we're done locking >[2018-08-18 10:05:04.030976] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-15 >[2018-08-18 10:05:04.030983] D [MSGID: 0] [afr-transaction.c:1857:afr_post_blocking_inodelk_cbk] 0-vol0-replicate-21: Blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.030985] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-16 >[2018-08-18 10:05:04.030994] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-17 >[2018-08-18 10:05:04.031003] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-18 >[2018-08-18 10:05:04.031059] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 10:05:04.031088] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-29: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.031112] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 10:05:04.031135] D [MSGID: 0] [afr-lk-common.c:1189:afr_lock_blocking] 0-vol0-replicate-20: we're done locking >[2018-08-18 10:05:04.031173] D [MSGID: 0] [afr-transaction.c:1857:afr_post_blocking_inodelk_cbk] 0-vol0-replicate-20: Blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.031213] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 10:05:04.031248] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-3: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.031263] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 10:05:04.031257] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-0: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.031281] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-27: lk op is for a transaction >[2018-08-18 10:05:04.031345] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-1: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.031345] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-30: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.031427] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 10:05:04.031416] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.031461] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.031460] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:04.031511] D [MSGID: 0] [afr-lk-common.c:1189:afr_lock_blocking] 0-vol0-replicate-22: we're done locking >[2018-08-18 10:05:04.031496] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.031529] D [MSGID: 0] [afr-transaction.c:1857:afr_post_blocking_inodelk_cbk] 0-vol0-replicate-22: Blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.031554] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:04.031579] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-32: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.031584] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 10:05:04.031615] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 10:05:04.031634] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-20: lk op is for a transaction >[2018-08-18 10:05:04.031655] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-12: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.031686] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 10:05:04.031721] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 10:05:04.031730] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-33: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.031741] D [MSGID: 0] [afr-lk-common.c:1189:afr_lock_blocking] 0-vol0-replicate-24: we're done locking >[2018-08-18 10:05:04.031750] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.031763] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 10:05:04.031770] D [MSGID: 0] [afr-transaction.c:1857:afr_post_blocking_inodelk_cbk] 0-vol0-replicate-24: Blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.031774] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-34: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.031770] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-30: lk op is for a transaction >[2018-08-18 10:05:04.031852] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.031886] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:04.031886] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-35: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.031946] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-22: lk op is for a transaction >[2018-08-18 10:05:04.031942] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 10:05:04.031979] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.031983] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-15: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.032016] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 10:05:04.032017] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 10:05:04.032025] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-5: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.032104] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-6: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.032093] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:04.032124] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 10:05:04.032108] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.032135] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 10:05:04.032152] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-17: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.032189] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.032223] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 10:05:04.032248] D [MSGID: 0] [afr-lk-common.c:1189:afr_lock_blocking] 0-vol0-replicate-26: we're done locking >[2018-08-18 10:05:04.032268] D [MSGID: 0] [afr-transaction.c:1857:afr_post_blocking_inodelk_cbk] 0-vol0-replicate-26: Blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.032262] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 10:05:04.032263] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-8: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.032309] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-18: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.032337] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 10:05:04.032348] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 10:05:04.032340] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-19: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.032341] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-9: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.032392] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 10:05:04.032399] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 10:05:04.032425] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 10:05:04.032475] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:04.032501] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-20: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.032512] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 10:05:04.032530] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-2: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.032537] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-26: lk op is for a transaction >[2018-08-18 10:05:04.032566] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.032620] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 10:05:04.032640] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-22: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.032646] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 10:05:04.032725] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 10:05:04.032735] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-23: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.032829] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.032862] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 10:05:04.032873] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 10:05:04.032883] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-22: lk op is for a transaction >[2018-08-18 10:05:04.032917] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:04.032945] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-25: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.032960] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-0: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.032987] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-23: lk op is for a transaction >[2018-08-18 10:05:04.033008] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-26: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.032986] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-2: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.033000] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-1: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.033051] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c4c0428c0, vol0-client-2 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.032987] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c4c0428c0, vol0-client-0 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.033065] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-20: lk op is for a transaction >[2018-08-18 10:05:04.033068] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c4c0428c0, vol0-client-1 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.033125] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 10:05:04.033207] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-27: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.033309] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-28: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.033343] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 10:05:04.033442] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-29: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.033552] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-3: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.033572] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-27: lk op is for a transaction >[2018-08-18 10:05:04.033604] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-30: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.033643] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.033674] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 10:05:04.033724] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:04.033783] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-26: lk op is for a transaction >[2018-08-18 10:05:04.033819] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-32: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.033851] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-33: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.033879] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 10:05:04.033932] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-34: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.033995] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-30: lk op is for a transaction >[2018-08-18 10:05:04.034018] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:04.034149] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 10:05:04.034153] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-35: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.034169] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 10:05:04.034172] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.034208] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 10:05:04.034235] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-5: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.034375] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-6: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.034403] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.034434] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 10:05:04.034451] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 10:05:04.034485] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 10:05:04.034552] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-9: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.034638] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 10:05:04.034660] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 10:05:04.034807] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 10:05:04.034928] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-8: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.035299] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 10:05:04.035420] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-1: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.035413] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-0: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.035444] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-2: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.035447] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c54077330, vol0-client-1 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.035463] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c54077330, vol0-client-0 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.035476] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c54077330, vol0-client-2 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.037188] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS on vol0-replicate-24 >[2018-08-18 10:05:04.037622] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS with op_ret 0 >[2018-08-18 10:05:04.041795] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt on vol0-replicate-34 >[2018-08-18 10:05:04.042163] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt with op_ret 0 >[2018-08-18 10:05:04.046108] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx on vol0-replicate-20 >[2018-08-18 10:05:04.046512] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx with op_ret 0 >[2018-08-18 10:05:04.050503] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0 on vol0-replicate-23 >[2018-08-18 10:05:04.050929] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0 with op_ret 0 >[2018-08-18 10:05:04.054915] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m on vol0-replicate-5 >[2018-08-18 10:05:04.055173] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055177] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055265] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055292] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055303] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m with op_ret -1 [No such file or directory] >[2018-08-18 10:05:04.055314] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m missing on subvol vol0-replicate-5 >[2018-08-18 10:05:04.055322] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:04.055559] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055618] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055638] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055679] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055675] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055693] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-0 >[2018-08-18 10:05:04.055703] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055746] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055766] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055808] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055801] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055855] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055859] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055862] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055879] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-2 >[2018-08-18 10:05:04.055861] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055899] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055908] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-1 >[2018-08-18 10:05:04.055912] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055918] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055916] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055931] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055938] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-3 >[2018-08-18 10:05:04.055952] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.055972] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-4 >[2018-08-18 10:05:04.055998] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056043] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056099] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056108] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056115] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056144] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056158] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-6 >[2018-08-18 10:05:04.056186] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056217] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056247] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056279] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056342] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056348] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056376] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056372] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056381] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056373] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056411] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056411] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056441] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-7 >[2018-08-18 10:05:04.056423] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-8 >[2018-08-18 10:05:04.056445] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056483] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-9 >[2018-08-18 10:05:04.056482] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056484] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056537] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056538] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056560] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056585] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056595] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-10 >[2018-08-18 10:05:04.056604] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056618] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-11 >[2018-08-18 10:05:04.056629] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056638] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056642] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056673] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056673] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056686] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-12 >[2018-08-18 10:05:04.056733] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056734] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056763] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056778] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056804] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056805] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-13 >[2018-08-18 10:05:04.056829] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056852] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056854] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056891] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-14 >[2018-08-18 10:05:04.056896] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056898] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056960] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.056983] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057013] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057054] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057054] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057082] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057068] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057116] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-17 >[2018-08-18 10:05:04.057089] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-15 >[2018-08-18 10:05:04.057149] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057152] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057181] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057183] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057204] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-16 >[2018-08-18 10:05:04.057269] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057271] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057302] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057284] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057335] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057337] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057363] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-18 >[2018-08-18 10:05:04.057368] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057396] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057403] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057433] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057433] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057474] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057506] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057512] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057544] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057545] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-21 >[2018-08-18 10:05:04.057547] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057555] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057592] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057600] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057613] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057597] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057621] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-19 >[2018-08-18 10:05:04.057642] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-20 >[2018-08-18 10:05:04.057645] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057671] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057673] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057689] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057703] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-22 >[2018-08-18 10:05:04.057714] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057721] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057738] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-25 >[2018-08-18 10:05:04.057747] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057757] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057765] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057787] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057784] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057786] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057795] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057799] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-26 >[2018-08-18 10:05:04.057804] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-23 >[2018-08-18 10:05:04.057816] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057829] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-24 >[2018-08-18 10:05:04.057866] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057942] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.057957] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058008] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058019] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058041] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058067] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058087] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-29 >[2018-08-18 10:05:04.058108] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058140] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058143] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058134] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058136] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058175] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058179] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058208] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-30 >[2018-08-18 10:05:04.058211] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058231] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058232] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058255] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-28 >[2018-08-18 10:05:04.058258] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-27 >[2018-08-18 10:05:04.058298] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058302] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058327] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058353] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058375] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058394] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058403] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-5 >[2018-08-18 10:05:04.058438] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058438] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058454] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058476] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058480] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058505] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058513] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058515] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-32 >[2018-08-18 10:05:04.058517] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058521] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-33 >[2018-08-18 10:05:04.058544] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058577] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058592] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-31 >[2018-08-18 10:05:04.058697] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058716] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058725] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-35 >[2018-08-18 10:05:04.058727] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058745] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058756] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-34 >[2018-08-18 10:05:04.058768] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-5 cached_subvol null >[2018-08-18 10:05:04.058777] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m >[2018-08-18 10:05:04.058786] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058797] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058808] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058819] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.058830] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c40cf80, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.059612] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0 returned with op_ret 0 >[2018-08-18 10:05:04.062458] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a on vol0-replicate-2 >[2018-08-18 10:05:04.062745] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.062765] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.062776] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 10:05:04.062759] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0 returned with op_ret 0" repeated 35 times between [2018-08-18 10:05:04.059612] and [2018-08-18 10:05:04.062491] >[2018-08-18 10:05:04.062775] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m on vol0-replicate-5 >[2018-08-18 10:05:04.062811] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.062839] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a with op_ret -1 [No such file or directory] >[2018-08-18 10:05:04.062861] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a missing on subvol vol0-replicate-2 >[2018-08-18 10:05:04.062886] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:04.063041] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063068] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063127] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063156] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063168] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m with op_ret -1 [No such file or directory] >[2018-08-18 10:05:04.063179] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m missing on subvol vol0-replicate-5 >[2018-08-18 10:05:04.063188] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:04.063210] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063222] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063305] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063320] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063351] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063381] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063381] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063392] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063415] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063436] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-1 >[2018-08-18 10:05:04.063453] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063459] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-0 >[2018-08-18 10:05:04.063497] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063502] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063514] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063529] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063553] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063540] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063572] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063590] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063606] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063608] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063612] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-0 >[2018-08-18 10:05:04.063619] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-3 >[2018-08-18 10:05:04.063613] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063620] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063633] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063676] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063671] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063694] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063704] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-4 >[2018-08-18 10:05:04.063694] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063711] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063695] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063707] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063740] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063748] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063762] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063750] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063754] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063800] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063795] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063805] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-1 >[2018-08-18 10:05:04.063793] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-2 >[2018-08-18 10:05:04.063816] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063807] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063831] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063845] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063849] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063847] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063857] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063864] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-5 >[2018-08-18 10:05:04.063863] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-3 >[2018-08-18 10:05:04.063879] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063886] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-4 >[2018-08-18 10:05:04.063907] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063916] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063927] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063966] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063991] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.063991] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064015] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-6 >[2018-08-18 10:05:04.064014] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064065] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064058] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064074] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064085] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064098] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-6 >[2018-08-18 10:05:04.064110] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064110] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064139] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064129] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064176] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064176] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064183] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064193] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-7 >[2018-08-18 10:05:04.064196] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064203] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064208] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064211] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064211] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064229] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064231] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-7 >[2018-08-18 10:05:04.064248] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-8 >[2018-08-18 10:05:04.064255] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064288] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064289] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064304] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064296] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064306] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064308] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064334] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064338] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064323] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064335] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064348] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064370] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-10 >[2018-08-18 10:05:04.064380] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064380] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-8 >[2018-08-18 10:05:04.064387] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-9 >[2018-08-18 10:05:04.064400] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-9 >[2018-08-18 10:05:04.064390] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064406] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064423] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064419] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064410] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064424] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064455] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064454] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064459] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064457] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064480] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064482] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064480] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-11 >[2018-08-18 10:05:04.064493] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064492] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-10 >[2018-08-18 10:05:04.064487] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064484] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-12 >[2018-08-18 10:05:04.064530] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064517] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064543] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-13 >[2018-08-18 10:05:04.064549] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064564] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064578] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064580] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064586] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-11 >[2018-08-18 10:05:04.064595] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064618] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064625] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064630] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064634] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064644] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064650] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064662] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064664] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064674] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064678] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-14 >[2018-08-18 10:05:04.064684] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-15 >[2018-08-18 10:05:04.064687] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-12 >[2018-08-18 10:05:04.064707] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064737] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064743] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064737] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064776] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064787] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064789] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064790] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064790] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064810] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-13 >[2018-08-18 10:05:04.064825] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064830] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064838] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064839] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064839] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-16 >[2018-08-18 10:05:04.064860] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064861] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064869] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064874] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-14 >[2018-08-18 10:05:04.064878] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064886] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064888] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064913] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-18 >[2018-08-18 10:05:04.064918] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064929] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064935] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064949] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064952] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064959] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-17 >[2018-08-18 10:05:04.064963] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-15 >[2018-08-18 10:05:04.064978] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.064981] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065000] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065001] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065009] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065019] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065019] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065051] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-19 >[2018-08-18 10:05:04.065053] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065064] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065075] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065093] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065097] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065104] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-20 >[2018-08-18 10:05:04.065111] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-16 >[2018-08-18 10:05:04.065115] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065124] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065143] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065145] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065177] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065186] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065199] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065213] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065226] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-21 >[2018-08-18 10:05:04.065222] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065232] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065258] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065280] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065281] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065282] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065284] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065290] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065307] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-22 >[2018-08-18 10:05:04.065307] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065301] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065313] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065318] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065328] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-23 >[2018-08-18 10:05:04.065315] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065353] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-18 >[2018-08-18 10:05:04.065357] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065359] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065377] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-17 >[2018-08-18 10:05:04.065402] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065424] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065427] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065434] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065440] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065435] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065450] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065471] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065480] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065485] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-19 >[2018-08-18 10:05:04.065485] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065502] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-24 >[2018-08-18 10:05:04.065504] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065524] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065541] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065549] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-26 >[2018-08-18 10:05:04.065544] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065562] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065564] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065592] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065600] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-20 >[2018-08-18 10:05:04.065602] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065608] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065637] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065642] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065661] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065661] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065664] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065678] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065691] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-21 >[2018-08-18 10:05:04.065675] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065692] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-25 >[2018-08-18 10:05:04.065717] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065738] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065742] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065765] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065742] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-27 >[2018-08-18 10:05:04.065758] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065762] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065769] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065754] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065766] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065825] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065823] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065823] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065835] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065824] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065848] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065848] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-23 >[2018-08-18 10:05:04.065856] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-22 >[2018-08-18 10:05:04.065867] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-28 >[2018-08-18 10:05:04.065868] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065876] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065884] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-29 >[2018-08-18 10:05:04.065885] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065920] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065912] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065942] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065944] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065960] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.065980] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-30 >[2018-08-18 10:05:04.065981] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066003] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066014] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066026] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066022] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066036] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066038] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066052] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066071] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-24 >[2018-08-18 10:05:04.066068] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066075] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066083] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-31 >[2018-08-18 10:05:04.066085] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066115] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-32 >[2018-08-18 10:05:04.066120] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066124] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066158] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066164] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066160] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066192] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066194] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-5 >[2018-08-18 10:05:04.066207] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-33 >[2018-08-18 10:05:04.066206] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066235] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066246] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066249] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066263] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066258] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066275] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-25 >[2018-08-18 10:05:04.066258] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066296] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066313] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066326] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066337] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066344] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-34 >[2018-08-18 10:05:04.066377] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066413] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066431] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066429] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066430] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066453] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066431] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066454] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066435] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066475] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066480] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066491] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-26 >[2018-08-18 10:05:04.066495] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m) from subvol vol0-replicate-35 >[2018-08-18 10:05:04.066499] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066512] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-5 cached_subvol null >[2018-08-18 10:05:04.066499] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066520] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m >[2018-08-18 10:05:04.066520] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-28 >[2018-08-18 10:05:04.066526] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066496] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066541] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066530] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066527] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-29 >[2018-08-18 10:05:04.066551] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066556] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-27 >[2018-08-18 10:05:04.066564] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066563] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066573] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066571] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066578] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066598] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066624] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-30 >[2018-08-18 10:05:04.066640] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066650] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066667] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066675] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-31 >[2018-08-18 10:05:04.066697] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066719] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066742] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066765] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066778] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-32 >[2018-08-18 10:05:04.066901] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-1': avail_percent is: 95.00 and avail_space is: 7649958682624 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.066902] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066910] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066929] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-2': avail_percent is: 96.00 and avail_space is: 7682660810752 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.066959] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-0': avail_percent is: 96.00 and avail_space is: 7683146379264 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.066959] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066995] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.066998] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-5': avail_percent is: 95.00 and avail_space is: 7674812334080 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067011] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-2 >[2018-08-18 10:05:04.067060] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-4': avail_percent is: 96.00 and avail_space is: 7680314966016 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067062] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.067072] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.067076] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.067064] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.067106] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.067110] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.067110] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.067129] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.067137] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-33 >[2018-08-18 10:05:04.067136] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-6': avail_percent is: 94.00 and avail_space is: 7572583333888 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067144] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-35 >[2018-08-18 10:05:04.067159] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.067171] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.067167] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-9': avail_percent is: 95.00 and avail_space is: 7675672170496 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067183] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a) from subvol vol0-replicate-34 >[2018-08-18 10:05:04.067201] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-2 cached_subvol null >[2018-08-18 10:05:04.067211] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a >[2018-08-18 10:05:04.067217] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.067213] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-3': avail_percent is: 96.00 and avail_space is: 7679949987840 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067224] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.067233] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.067240] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.067249] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3404e2f0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.067245] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-7': avail_percent is: 95.00 and avail_space is: 7602560430080 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067273] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-11': avail_percent is: 95.00 and avail_space is: 7666756444160 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067298] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-10': avail_percent is: 94.00 and avail_space is: 7587978166272 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067319] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-8': avail_percent is: 95.00 and avail_space is: 7674545618944 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067380] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-13': avail_percent is: 95.00 and avail_space is: 7672180158464 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067431] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-12': avail_percent is: 95.00 and avail_space is: 7662852456448 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067451] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-15': avail_percent is: 95.00 and avail_space is: 7657355005952 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067582] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-18': avail_percent is: 95.00 and avail_space is: 7620654678016 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067639] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-16': avail_percent is: 96.00 and avail_space is: 7681138819072 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067645] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-0': avail_percent is: 96.00 and avail_space is: 7683146379264 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067682] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-19': avail_percent is: 95.00 and avail_space is: 7661031256064 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067685] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-1': avail_percent is: 95.00 and avail_space is: 7649958682624 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067715] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-21': avail_percent is: 95.00 and avail_space is: 7663950176256 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067722] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-2': avail_percent is: 96.00 and avail_space is: 7682660810752 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067745] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-20': avail_percent is: 95.00 and avail_space is: 7611650596864 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067776] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-17': avail_percent is: 95.00 and avail_space is: 7670394257408 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067815] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-4': avail_percent is: 96.00 and avail_space is: 7680314966016 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067843] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-6': avail_percent is: 94.00 and avail_space is: 7572583333888 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067849] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-22': avail_percent is: 95.00 and avail_space is: 7668892258304 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067874] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-7': avail_percent is: 95.00 and avail_space is: 7602560430080 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067878] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-25': avail_percent is: 95.00 and avail_space is: 7650944450560 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067921] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-23': avail_percent is: 95.00 and avail_space is: 7670241071104 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067913] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-5': avail_percent is: 95.00 and avail_space is: 7674812334080 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067950] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-8': avail_percent is: 95.00 and avail_space is: 7674545618944 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067980] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-24': avail_percent is: 95.00 and avail_space is: 7665744367616 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.067997] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-3': avail_percent is: 96.00 and avail_space is: 7679949987840 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068020] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-26': avail_percent is: 95.00 and avail_space is: 7659003916288 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068049] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-10': avail_percent is: 94.00 and avail_space is: 7587978166272 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068069] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-30': avail_percent is: 95.00 and avail_space is: 7677999882240 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068082] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-9': avail_percent is: 95.00 and avail_space is: 7675672170496 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068109] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-31': avail_percent is: 95.00 and avail_space is: 7678388379648 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068139] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-29': avail_percent is: 95.00 and avail_space is: 7639891890176 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068166] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-33': avail_percent is: 95.00 and avail_space is: 7679374594048 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068181] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-12': avail_percent is: 95.00 and avail_space is: 7662852456448 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068189] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-27': avail_percent is: 95.00 and avail_space is: 7652761243648 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068209] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-14': avail_percent is: 95.00 and avail_space is: 7671006871552 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068218] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-11': avail_percent is: 95.00 and avail_space is: 7666756444160 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068230] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-32': avail_percent is: 95.00 and avail_space is: 7664158932992 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068244] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-13': avail_percent is: 95.00 and avail_space is: 7672180158464 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068253] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-34': avail_percent is: 95.00 and avail_space is: 7668154388480 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068274] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-14': avail_percent is: 95.00 and avail_space is: 7671006871552 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068309] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-28': avail_percent is: 95.00 and avail_space is: 7652597972992 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068348] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-35': avail_percent is: 95.00 and avail_space is: 7666729115648 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068416] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-15': avail_percent is: 95.00 and avail_space is: 7657355005952 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068444] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-17': avail_percent is: 95.00 and avail_space is: 7670394257408 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068468] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-16': avail_percent is: 96.00 and avail_space is: 7681138819072 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068504] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-18': avail_percent is: 95.00 and avail_space is: 7620654678016 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068526] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-21': avail_percent is: 95.00 and avail_space is: 7663950176256 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068557] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-20': avail_percent is: 95.00 and avail_space is: 7611650596864 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068587] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-19': avail_percent is: 95.00 and avail_space is: 7661031256064 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068608] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-16: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.068613] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-22': avail_percent is: 95.00 and avail_space is: 7668892258304 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068610] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-15: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.068632] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c54000d90, vol0-client-15 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.068634] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c54000d90, vol0-client-16 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.068643] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-23': avail_percent is: 95.00 and avail_space is: 7670241071104 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068670] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-24': avail_percent is: 95.00 and avail_space is: 7665744367616 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068684] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-17: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:04.068701] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c54000d90, vol0-client-17 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:04.068709] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 10:05:04.068711] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-25': avail_percent is: 95.00 and avail_space is: 7650944450560 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068716] D [MSGID: 0] [afr-transaction.c:1928:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-5: Non blocking entrylks failed. Proceeding to blocking >[2018-08-18 10:05:04.068773] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-26': avail_percent is: 95.00 and avail_space is: 7659003916288 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068820] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-27': avail_percent is: 95.00 and avail_space is: 7652761243648 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068850] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-28': avail_percent is: 95.00 and avail_space is: 7652597972992 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068870] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-29': avail_percent is: 95.00 and avail_space is: 7639891890176 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068911] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-30': avail_percent is: 95.00 and avail_space is: 7677999882240 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.068986] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-31': avail_percent is: 95.00 and avail_space is: 7678388379648 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.069014] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-32': avail_percent is: 95.00 and avail_space is: 7664158932992 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.069055] D [MSGID: 0] [dht-common.c:6923:dht_create_wind_to_avail_subvol] 0-vol0-dht: creating /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a on vol0-replicate-2 >[2018-08-18 10:05:04.069093] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-33': avail_percent is: 95.00 and avail_space is: 7679374594048 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.069123] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-34': avail_percent is: 95.00 and avail_space is: 7668154388480 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.069159] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-35': avail_percent is: 95.00 and avail_space is: 7666729115648 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.069288] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-2: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:04.075275] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a on vol0-replicate-10 >[2018-08-18 10:05:04.075541] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.075567] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.075572] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.075601] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.075612] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a with op_ret -1 [No such file or directory] >[2018-08-18 10:05:04.075621] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a missing on subvol vol0-replicate-10 >[2018-08-18 10:05:04.075629] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:04.075916] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.075925] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.075918] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.075931] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.075974] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.075975] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.075984] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076027] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076029] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-0 >[2018-08-18 10:05:04.076039] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076040] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076054] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076066] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-1 >[2018-08-18 10:05:04.076079] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076085] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076086] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076104] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-2 >[2018-08-18 10:05:04.076135] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076144] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076169] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076184] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076184] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-3 >[2018-08-18 10:05:04.076210] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076242] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076246] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076255] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-4 >[2018-08-18 10:05:04.076263] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076296] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076311] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076332] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076344] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076350] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-5 >[2018-08-18 10:05:04.076376] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076420] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076420] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076435] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076466] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-6 >[2018-08-18 10:05:04.076467] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076497] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076515] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-7 >[2018-08-18 10:05:04.076522] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076586] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076583] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076643] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076649] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076650] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076666] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076689] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076694] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-8 >[2018-08-18 10:05:04.076707] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076690] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076726] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076731] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-9 >[2018-08-18 10:05:04.076741] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076793] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076811] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076812] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076834] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-11 >[2018-08-18 10:05:04.076869] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076872] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076883] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076902] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076904] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076928] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-12 >[2018-08-18 10:05:04.076931] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076965] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.076978] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-13 >[2018-08-18 10:05:04.077001] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077025] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077058] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077077] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077091] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-14 >[2018-08-18 10:05:04.077124] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077129] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077133] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077175] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077180] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077202] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-15 >[2018-08-18 10:05:04.077209] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077240] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077235] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077259] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077272] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-16 >[2018-08-18 10:05:04.077291] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077312] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077332] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077335] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077355] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-17 >[2018-08-18 10:05:04.077372] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077414] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077444] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077456] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077463] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077498] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077501] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077517] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-18 >[2018-08-18 10:05:04.077525] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-19 >[2018-08-18 10:05:04.077528] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077518] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077568] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077585] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077576] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077603] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077627] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077631] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077632] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077663] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-20 >[2018-08-18 10:05:04.077671] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-21 >[2018-08-18 10:05:04.077673] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077722] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077753] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077774] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077788] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-22 >[2018-08-18 10:05:04.077791] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077824] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077821] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077813] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077860] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077865] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077857] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077894] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077896] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077913] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-24 >[2018-08-18 10:05:04.077921] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-23 >[2018-08-18 10:05:04.077929] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077941] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.077989] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078019] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078038] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-26 >[2018-08-18 10:05:04.078065] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078068] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078075] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078109] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078115] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078117] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078136] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078111] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078133] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078150] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-27 >[2018-08-18 10:05:04.078158] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-25 >[2018-08-18 10:05:04.078180] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078195] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-28 >[2018-08-18 10:05:04.078217] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078212] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078253] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078257] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078259] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078258] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078288] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078289] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078311] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-29 >[2018-08-18 10:05:04.078316] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-30 >[2018-08-18 10:05:04.078321] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078339] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078352] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078369] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078379] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-31 >[2018-08-18 10:05:04.078411] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078473] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078499] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078509] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078511] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078529] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078547] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078558] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-32 >[2018-08-18 10:05:04.078568] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078581] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-33 >[2018-08-18 10:05:04.078560] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078608] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078611] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078622] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-10 >[2018-08-18 10:05:04.078642] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078639] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078663] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078677] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-34 >[2018-08-18 10:05:04.078683] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078697] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078714] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078733] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078749] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a) from subvol vol0-replicate-35 >[2018-08-18 10:05:04.078760] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-10 cached_subvol null >[2018-08-18 10:05:04.078770] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a >[2018-08-18 10:05:04.078777] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078786] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078798] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078807] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.078818] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c0336b0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.079155] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-0': avail_percent is: 96.00 and avail_space is: 7683146379264 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.079183] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-1': avail_percent is: 95.00 and avail_space is: 7649958682624 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.079209] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-2': avail_percent is: 96.00 and avail_space is: 7682660810752 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.079239] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-3': avail_percent is: 96.00 and avail_space is: 7679949987840 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.079310] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-4': avail_percent is: 96.00 and avail_space is: 7680314966016 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.079357] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-5': avail_percent is: 95.00 and avail_space is: 7674812334080 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.079389] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-6': avail_percent is: 94.00 and avail_space is: 7572583333888 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.079423] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-7': avail_percent is: 95.00 and avail_space is: 7602560430080 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.079444] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-8': avail_percent is: 95.00 and avail_space is: 7674545618944 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.079496] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-9': avail_percent is: 95.00 and avail_space is: 7675672170496 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.079557] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-10': avail_percent is: 94.00 and avail_space is: 7587978166272 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.079619] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-11': avail_percent is: 95.00 and avail_space is: 7666756444160 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.079663] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-12': avail_percent is: 95.00 and avail_space is: 7662852456448 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.079699] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-13': avail_percent is: 95.00 and avail_space is: 7672180158464 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.079720] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-14': avail_percent is: 95.00 and avail_space is: 7671006871552 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.079783] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-15': avail_percent is: 95.00 and avail_space is: 7657355005952 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.079848] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-16': avail_percent is: 96.00 and avail_space is: 7681138819072 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.079871] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-17': avail_percent is: 95.00 and avail_space is: 7670394257408 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.079933] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-18': avail_percent is: 95.00 and avail_space is: 7620654678016 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.080037] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-21': avail_percent is: 95.00 and avail_space is: 7663950176256 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.080084] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-22': avail_percent is: 95.00 and avail_space is: 7668892258304 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.080104] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-19': avail_percent is: 95.00 and avail_space is: 7661031256064 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.080138] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-20': avail_percent is: 95.00 and avail_space is: 7611650596864 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.080182] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-23': avail_percent is: 95.00 and avail_space is: 7670241071104 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.080220] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-24': avail_percent is: 95.00 and avail_space is: 7665744367616 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.080244] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-25': avail_percent is: 95.00 and avail_space is: 7650944450560 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.080268] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-26': avail_percent is: 95.00 and avail_space is: 7659003916288 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.080341] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-27': avail_percent is: 95.00 and avail_space is: 7652761243648 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.080376] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-28': avail_percent is: 95.00 and avail_space is: 7652597972992 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.080405] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-29': avail_percent is: 95.00 and avail_space is: 7639891890176 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.080464] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-30': avail_percent is: 95.00 and avail_space is: 7677999882240 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.080501] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-31': avail_percent is: 95.00 and avail_space is: 7678388379648 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.080538] D [MSGID: 0] [dht-common.c:6923:dht_create_wind_to_avail_subvol] 0-vol0-dht: creating /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a on vol0-replicate-10 >[2018-08-18 10:05:04.080563] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-32': avail_percent is: 95.00 and avail_space is: 7664158932992 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.080600] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-33': avail_percent is: 95.00 and avail_space is: 7679374594048 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.080638] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-34': avail_percent is: 95.00 and avail_space is: 7668154388480 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.080660] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-35': avail_percent is: 95.00 and avail_space is: 7666729115648 and avail_inodes is: 99.00 >[2018-08-18 10:05:04.080891] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-10: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:04.095487] D [MSGID: 0] [afr-lk-common.c:1189:afr_lock_blocking] 0-vol0-replicate-5: we're done locking >[2018-08-18 10:05:04.095510] D [MSGID: 0] [afr-transaction.c:1908:afr_post_blocking_entrylk_cbk] 0-vol0-replicate-5: Blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:04.095695] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-16: remote operation failed. Path: /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m [File exists] >[2018-08-18 10:05:04.095704] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-17: remote operation failed. Path: /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m [File exists] >[2018-08-18 10:05:04.095691] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-15: remote operation failed. Path: /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m [File exists] >[2018-08-18 10:05:04.095736] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c54000d90, vol0-client-16 returned -1 error: File exists [File exists] >[2018-08-18 10:05:04.095731] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c54000d90, vol0-client-17 returned -1 error: File exists [File exists] >[2018-08-18 10:05:04.095751] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c54000d90, vol0-client-15 returned -1 error: File exists [File exists] >[2018-08-18 10:05:04.095780] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 10:05:04.095993] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-replicate-5 returned -1 error: File exists [File exists] >[2018-08-18 10:05:04.096084] D [MSGID: 0] [dht-common.c:7688:dht_mkdir_hashed_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-dht returned -1 error: File exists [File exists] >[2018-08-18 10:05:04.096098] D [MSGID: 0] [md-cache.c:1410:mdc_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0-md-cache returned -1 error: File exists [File exists] >[2018-08-18 10:05:04.096107] D [MSGID: 0] [io-stats.c:2247:io_stats_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c2c415070, vol0 returned -1 error: File exists [File exists] >[2018-08-18 10:05:04.099707] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-2 >[2018-08-18 10:05:04.099768] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 10:05:04.106501] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-10 >[2018-08-18 10:05:04.106574] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 10:05:04.287079] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7f3c3c013750) as it exceeds the maximum available buffer size >[2018-08-18 10:05:04.288954] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.288976] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=d8824496-d2df-466a-818c-209a2e4f09aa, gen=0): picking the request for winding >[2018-08-18 10:05:04.288988] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=d8824496-d2df-466a-818c-209a2e4f09aa, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:04.289289] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.306567] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=d8824496-d2df-466a-818c-209a2e4f09aa, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:04.306733] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=d8824496-d2df-466a-818c-209a2e4f09aa, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:04.306895] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.360579] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7f3c34075ea0) as it exceeds the maximum available buffer size >[2018-08-18 10:05:04.362116] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.362142] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=a0aa13d4-bc23-45a2-9478-e6420c92b07e, gen=0): picking the request for winding >[2018-08-18 10:05:04.362154] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=a0aa13d4-bc23-45a2-9478-e6420c92b07e, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:04.362542] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-2: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.380149] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=a0aa13d4-bc23-45a2-9478-e6420c92b07e, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:04.380269] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=a0aa13d4-bc23-45a2-9478-e6420c92b07e, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:04.380377] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.542215] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7f3c3c050670) as it exceeds the maximum available buffer size >[2018-08-18 10:05:04.545633] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.545666] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=d8824496-d2df-466a-818c-209a2e4f09aa, gen=0): picking the request for winding >[2018-08-18 10:05:04.545683] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=d8824496-d2df-466a-818c-209a2e4f09aa, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:04.545764] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.545782] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 10:05:04.545816] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7f3c3c013750) allocated with standard calloc() >[2018-08-18 10:05:04.559824] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.559860] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=1, gfid=d8824496-d2df-466a-818c-209a2e4f09aa. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=d8824496-d2df-466a-818c-209a2e4f09aa, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:04.561001] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7f3c34019f40) as it exceeds the maximum available buffer size >[2018-08-18 10:05:04.562231] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=d8824496-d2df-466a-818c-209a2e4f09aa, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:04.562386] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=d8824496-d2df-466a-818c-209a2e4f09aa, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:04.562506] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.562522] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=d8824496-d2df-466a-818c-209a2e4f09aa, gen=1): picking the request for winding >[2018-08-18 10:05:04.562776] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=d8824496-d2df-466a-818c-209a2e4f09aa, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:04.562899] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 10:05:04.563083] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7f3c3c050670) allocated with standard calloc() >[2018-08-18 10:05:04.563537] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.563557] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=a0aa13d4-bc23-45a2-9478-e6420c92b07e, gen=0): picking the request for winding >[2018-08-18 10:05:04.563568] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=a0aa13d4-bc23-45a2-9478-e6420c92b07e, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:04.563628] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-2: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.563642] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 10:05:04.563664] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7f3c34075ea0) allocated with standard calloc() >[2018-08-18 10:05:04.564306] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002 on vol0-replicate-29 >[2018-08-18 10:05:04.564728] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.564782] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.564874] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.564913] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.564929] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002 with op_ret -1 [No such file or directory] >[2018-08-18 10:05:04.564943] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002 missing on subvol vol0-replicate-29 >[2018-08-18 10:05:04.564954] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:04.565261] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565305] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565339] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565395] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565410] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565425] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565430] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565448] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565462] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565469] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-0 >[2018-08-18 10:05:04.565480] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-1 >[2018-08-18 10:05:04.565464] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565500] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565543] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565542] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565584] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565579] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565617] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-2 >[2018-08-18 10:05:04.565630] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565649] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-3 >[2018-08-18 10:05:04.565666] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565675] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565705] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565720] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565720] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565748] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-4 >[2018-08-18 10:05:04.565749] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565793] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565805] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565815] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-5 >[2018-08-18 10:05:04.565847] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565871] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565906] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565922] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565932] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-6 >[2018-08-18 10:05:04.565979] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.565992] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566121] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566129] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566118] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566163] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566164] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566190] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566197] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566199] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566237] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-7 >[2018-08-18 10:05:04.566238] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566239] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566239] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566268] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-9 >[2018-08-18 10:05:04.566272] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-8 >[2018-08-18 10:05:04.566268] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566312] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566330] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566340] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-10 >[2018-08-18 10:05:04.566382] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566455] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566458] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566475] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566508] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566531] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566542] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-12 >[2018-08-18 10:05:04.566572] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566578] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566595] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566605] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-11 >[2018-08-18 10:05:04.566660] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566696] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566720] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566732] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-13 >[2018-08-18 10:05:04.566781] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566789] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566783] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566819] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566834] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566842] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566881] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566901] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566908] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566909] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566924] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566933] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-14 >[2018-08-18 10:05:04.566940] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-15 >[2018-08-18 10:05:04.566963] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566969] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566978] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.566988] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-16 >[2018-08-18 10:05:04.567005] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567019] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567029] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-17 >[2018-08-18 10:05:04.567087] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567113] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567160] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567194] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567199] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567216] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-18 >[2018-08-18 10:05:04.567217] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567244] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567279] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567284] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567301] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567304] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-19 >[2018-08-18 10:05:04.567345] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567384] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567399] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567400] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-20 >[2018-08-18 10:05:04.567444] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567441] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567476] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567502] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-21 >[2018-08-18 10:05:04.567506] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567512] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567544] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567553] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567564] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567574] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-22 >[2018-08-18 10:05:04.567588] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567591] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567611] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567623] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-23 >[2018-08-18 10:05:04.567649] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567675] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567719] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567753] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567767] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567770] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567770] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-24 >[2018-08-18 10:05:04.567813] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567826] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567834] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-25 >[2018-08-18 10:05:04.567870] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567901] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567921] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567931] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.567938] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-26 >[2018-08-18 10:05:04.567991] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568008] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568075] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568103] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568115] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-27 >[2018-08-18 10:05:04.568159] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568159] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568196] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568199] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568265] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568288] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568326] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568338] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568370] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568414] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-30 >[2018-08-18 10:05:04.568417] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568423] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568467] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-28 >[2018-08-18 10:05:04.568470] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568472] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568504] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568513] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-31 >[2018-08-18 10:05:04.568561] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568587] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568598] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568619] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568634] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568640] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-29 >[2018-08-18 10:05:04.568647] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568665] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568683] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-32 >[2018-08-18 10:05:04.568709] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568712] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568744] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568749] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568751] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568791] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-33 >[2018-08-18 10:05:04.568795] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568821] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568835] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-34 >[2018-08-18 10:05:04.568855] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568866] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568874] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568902] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350e570, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568912] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-35 >[2018-08-18 10:05:04.568922] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-29 cached_subvol null >[2018-08-18 10:05:04.568929] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002 >[2018-08-18 10:05:04.568936] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x350e570, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568943] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568959] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568968] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.568976] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x350e570, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.570371] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m returned with op_ret 0 >[2018-08-18 10:05:04.573580] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.573608] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m returned with op_ret 0" repeated 35 times between [2018-08-18 10:05:04.570371] and [2018-08-18 10:05:04.573186] >[2018-08-18 10:05:04.573608] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=d8824496-d2df-466a-818c-209a2e4f09aa, gen=0): picking the request for winding >[2018-08-18 10:05:04.573654] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a at vol0-replicate-10 >[2018-08-18 10:05:04.573818] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.573840] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=1, gfid=a0aa13d4-bc23-45a2-9478-e6420c92b07e. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=a0aa13d4-bc23-45a2-9478-e6420c92b07e, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:04.573910] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=d8824496-d2df-466a-818c-209a2e4f09aa, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:04.574542] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a returned with op_ret 0 >[2018-08-18 10:05:04.574723] D [md-cache.c:503:mdc_inode_iatt_set_validate] (-->/usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3cd0) [0x7f3d02157cd0] -->/usr/lib64/glusterfs/3.10.12/xlator/performance/md-cache.so(+0xa802) [0x7f3d01d3e802] -->/usr/lib64/glusterfs/3.10.12/xlator/performance/md-cache.so(+0x66cb) [0x7f3d01d3a6cb] ) 0-vol0-md-cache: discarding the iatt validate request(ctime_nsec) [Invalid argument] >[2018-08-18 10:05:04.574932] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002 on vol0-replicate-29 >[2018-08-18 10:05:04.575795] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.575952] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.575954] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.575991] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576007] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002 with op_ret -1 [No such file or directory] >[2018-08-18 10:05:04.576022] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002 missing on subvol vol0-replicate-29 >[2018-08-18 10:05:04.576054] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:04.576457] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576509] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576538] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576542] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576556] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576595] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576615] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-0 >[2018-08-18 10:05:04.576637] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576647] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576662] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576689] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576690] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576700] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576699] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576705] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-2 >[2018-08-18 10:05:04.576775] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576778] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576788] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576809] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-1 >[2018-08-18 10:05:04.576783] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576833] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576850] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576859] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576883] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576886] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576900] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-4 >[2018-08-18 10:05:04.576912] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-3 >[2018-08-18 10:05:04.576926] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576930] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576935] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576966] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576959] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576993] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.576993] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-5 >[2018-08-18 10:05:04.577012] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-6 >[2018-08-18 10:05:04.577027] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577041] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577077] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577104] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577126] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-7 >[2018-08-18 10:05:04.577163] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577207] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577250] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577275] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577277] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577299] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577301] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577314] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-8 >[2018-08-18 10:05:04.577329] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577341] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-9 >[2018-08-18 10:05:04.577342] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577410] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577411] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577429] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577437] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-10 >[2018-08-18 10:05:04.577507] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577536] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577541] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577552] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577542] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577536] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577573] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577568] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577598] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-12 >[2018-08-18 10:05:04.577609] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577617] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577625] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577626] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-11 >[2018-08-18 10:05:04.577647] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577658] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-13 >[2018-08-18 10:05:04.577657] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577690] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577689] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577735] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577755] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-14 >[2018-08-18 10:05:04.577756] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577775] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577846] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577867] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577883] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577891] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577901] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577919] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-15 >[2018-08-18 10:05:04.577919] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577941] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-16 >[2018-08-18 10:05:04.577977] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.577992] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578035] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578040] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578053] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578086] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578089] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578111] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578115] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578121] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578111] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-17 >[2018-08-18 10:05:04.578141] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-18 >[2018-08-18 10:05:04.578187] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578224] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578241] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-19 >[2018-08-18 10:05:04.578246] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578248] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578248] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578309] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578309] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578333] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578342] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-20 >[2018-08-18 10:05:04.578399] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578412] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578421] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578442] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578447] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578454] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-21 >[2018-08-18 10:05:04.578476] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578478] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578501] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578503] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-22 >[2018-08-18 10:05:04.578507] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578532] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578542] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-23 >[2018-08-18 10:05:04.578601] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578607] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578623] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578668] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578678] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578684] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578686] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-24 >[2018-08-18 10:05:04.578699] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578718] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578728] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-25 >[2018-08-18 10:05:04.578772] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578789] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578810] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578836] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578837] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578851] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-26 >[2018-08-18 10:05:04.578855] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578910] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578937] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578917] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578970] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578976] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.578988] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579006] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-28 >[2018-08-18 10:05:04.579003] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579018] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-27 >[2018-08-18 10:05:04.579023] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579097] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579098] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579142] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579178] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579182] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579184] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579180] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579226] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-30 >[2018-08-18 10:05:04.579232] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579233] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579252] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-31 >[2018-08-18 10:05:04.579260] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579287] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579318] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579338] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-32 >[2018-08-18 10:05:04.579340] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579379] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579392] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579411] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579420] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-33 >[2018-08-18 10:05:04.579445] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579439] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579486] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579483] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579504] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-29 >[2018-08-18 10:05:04.579524] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579540] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579555] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579561] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579565] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579609] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-34 >[2018-08-18 10:05:04.579611] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579633] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002) from subvol vol0-replicate-35 >[2018-08-18 10:05:04.579650] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-29 cached_subvol null >[2018-08-18 10:05:04.579661] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002 >[2018-08-18 10:05:04.579672] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579685] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579699] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579711] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579722] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3c09bd30, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.579798] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=a0aa13d4-bc23-45a2-9478-e6420c92b07e, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:04.579863] I [MSGID: 109066] [dht-rename.c:1610:dht_rename] 0-vol0-dht: renaming /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002_2ca8deaaa2ce11e89855246e9631871a (hash=vol0-replicate-10/cache=vol0-replicate-10) => /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002 (hash=vol0-replicate-29/cache=<nul>) >[2018-08-18 10:05:04.579927] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=a0aa13d4-bc23-45a2-9478-e6420c92b07e, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:04.580069] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.580087] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=a0aa13d4-bc23-45a2-9478-e6420c92b07e, gen=1): picking the request for winding >[2018-08-18 10:05:04.580297] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=a0aa13d4-bc23-45a2-9478-e6420c92b07e, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:04.580477] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 10:05:04.580606] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-29: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:04.580724] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7f3c34019f40) allocated with standard calloc() >[2018-08-18 10:05:04.581545] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006 on vol0-replicate-28 >[2018-08-18 10:05:04.581855] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.581866] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.581903] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.581938] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.581951] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006 with op_ret -1 [No such file or directory] >[2018-08-18 10:05:04.581963] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006 missing on subvol vol0-replicate-28 >[2018-08-18 10:05:04.581977] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:04.582292] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582309] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582334] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582360] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582371] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-0 >[2018-08-18 10:05:04.582409] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582433] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582439] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582484] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582489] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582511] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-1 >[2018-08-18 10:05:04.582539] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582540] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582576] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582579] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582601] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-2 >[2018-08-18 10:05:04.582623] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582666] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582710] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582716] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582716] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582740] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-3 >[2018-08-18 10:05:04.582767] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582798] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582805] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582823] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-4 >[2018-08-18 10:05:04.582840] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582871] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582892] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582902] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-5 >[2018-08-18 10:05:04.582964] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.582979] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583047] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583019] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583100] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583113] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583120] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583142] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-6 >[2018-08-18 10:05:04.583163] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583173] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583183] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-7 >[2018-08-18 10:05:04.583210] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583270] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583284] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583319] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583328] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583335] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-8 >[2018-08-18 10:05:04.583346] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583373] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583396] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-9 >[2018-08-18 10:05:04.583402] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583440] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583442] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583482] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583485] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583496] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-10 >[2018-08-18 10:05:04.583538] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583548] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583575] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583586] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-11 >[2018-08-18 10:05:04.583588] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583628] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583643] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583661] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583670] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-12 >[2018-08-18 10:05:04.583730] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583734] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583735] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583768] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583778] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583778] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-13 >[2018-08-18 10:05:04.583806] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583829] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583847] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583857] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-14 >[2018-08-18 10:05:04.583890] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583891] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583912] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583922] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.583930] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-15 >[2018-08-18 10:05:04.583998] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584008] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584041] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584072] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584086] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-16 >[2018-08-18 10:05:04.584112] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584113] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584116] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584152] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584154] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584179] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-17 >[2018-08-18 10:05:04.584191] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584219] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584236] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584236] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584268] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-18 >[2018-08-18 10:05:04.584265] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584274] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584305] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584311] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584332] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-19 >[2018-08-18 10:05:04.584334] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584357] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584381] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584401] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-20 >[2018-08-18 10:05:04.584408] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584417] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584471] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584481] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584503] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584514] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584524] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584525] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-21 >[2018-08-18 10:05:04.584547] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584561] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-22 >[2018-08-18 10:05:04.584580] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584584] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584647] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584651] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584669] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584678] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-23 >[2018-08-18 10:05:04.584712] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584737] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584753] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584758] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584784] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584797] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-24 >[2018-08-18 10:05:04.584804] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584826] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584844] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584844] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584856] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584873] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-25 >[2018-08-18 10:05:04.584884] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584896] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-26 >[2018-08-18 10:05:04.584938] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584941] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584941] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584974] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.584993] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-27 >[2018-08-18 10:05:04.585016] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585022] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585104] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585130] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585160] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585182] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585194] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-29 >[2018-08-18 10:05:04.585214] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585224] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585248] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585257] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585277] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585292] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-30 >[2018-08-18 10:05:04.585300] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585311] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585338] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585339] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585364] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585374] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585364] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-31 >[2018-08-18 10:05:04.585400] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585412] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-32 >[2018-08-18 10:05:04.585439] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585460] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585466] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585491] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585525] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-33 >[2018-08-18 10:05:04.585527] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585552] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585581] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585583] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585602] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585624] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-34 >[2018-08-18 10:05:04.585627] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585649] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-28 >[2018-08-18 10:05:04.585679] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585835] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585882] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585902] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x350c750, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585913] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-35 >[2018-08-18 10:05:04.585927] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-28 cached_subvol null >[2018-08-18 10:05:04.585938] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006 >[2018-08-18 10:05:04.585949] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x350c750, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585960] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585974] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.585987] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.586000] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x350c750, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.586623] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m returned with op_ret 0 >[2018-08-18 10:05:04.589644] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.589666] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m returned with op_ret 0" repeated 35 times between [2018-08-18 10:05:04.586623] and [2018-08-18 10:05:04.589308] >[2018-08-18 10:05:04.589665] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=a0aa13d4-bc23-45a2-9478-e6420c92b07e, gen=0): picking the request for winding >[2018-08-18 10:05:04.589698] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a at vol0-replicate-2 >[2018-08-18 10:05:04.589893] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=a0aa13d4-bc23-45a2-9478-e6420c92b07e, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:04.590016] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a returned with op_ret 0 >[2018-08-18 10:05:04.590133] D [md-cache.c:503:mdc_inode_iatt_set_validate] (-->/usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3cd0) [0x7f3d02157cd0] -->/usr/lib64/glusterfs/3.10.12/xlator/performance/md-cache.so(+0xa802) [0x7f3d01d3e802] -->/usr/lib64/glusterfs/3.10.12/xlator/performance/md-cache.so(+0x66cb) [0x7f3d01d3a6cb] ) 0-vol0-md-cache: discarding the iatt validate request(ctime_nsec) [Invalid argument] >[2018-08-18 10:05:04.590261] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006 on vol0-replicate-28 >[2018-08-18 10:05:04.590503] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.590505] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.590513] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.590554] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.590566] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006 with op_ret -1 [No such file or directory] >[2018-08-18 10:05:04.590577] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006 missing on subvol vol0-replicate-28 >[2018-08-18 10:05:04.590585] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:04.590859] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.590894] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.590902] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.590924] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.590939] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-0 >[2018-08-18 10:05:04.590950] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.590941] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.590980] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.590986] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.590993] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591004] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591035] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591038] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591060] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591073] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-1 >[2018-08-18 10:05:04.591074] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591080] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-2 >[2018-08-18 10:05:04.591081] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591108] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591123] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591146] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591168] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-3 >[2018-08-18 10:05:04.591195] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591216] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591231] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-4 >[2018-08-18 10:05:04.591233] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591253] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591267] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591282] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591294] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-5 >[2018-08-18 10:05:04.591355] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591371] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591396] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591436] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591456] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-6 >[2018-08-18 10:05:04.591473] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591517] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591533] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591531] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591568] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591572] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591603] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591609] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591618] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-7 >[2018-08-18 10:05:04.591622] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591645] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591653] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591664] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-8 >[2018-08-18 10:05:04.591687] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591704] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-9 >[2018-08-18 10:05:04.591712] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591740] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591740] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591771] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591767] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591802] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591808] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591823] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591831] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-10 >[2018-08-18 10:05:04.591855] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-11 >[2018-08-18 10:05:04.591856] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591851] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591889] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591910] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591923] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-12 >[2018-08-18 10:05:04.591978] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591995] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.591995] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592061] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592078] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-13 >[2018-08-18 10:05:04.592088] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592093] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592125] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592146] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592160] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-14 >[2018-08-18 10:05:04.592167] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592189] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592192] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592209] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592218] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-15 >[2018-08-18 10:05:04.592259] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592268] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592282] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592304] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592318] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-16 >[2018-08-18 10:05:04.592399] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592417] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592424] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592450] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592458] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592466] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-17 >[2018-08-18 10:05:04.592484] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592520] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592542] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592551] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-18 >[2018-08-18 10:05:04.592591] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592606] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592602] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592635] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592645] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-19 >[2018-08-18 10:05:04.592653] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592712] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592731] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592747] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592755] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-20 >[2018-08-18 10:05:04.592792] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592799] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592812] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592826] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592832] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-21 >[2018-08-18 10:05:04.592884] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592889] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592903] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592921] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.592931] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-22 >[2018-08-18 10:05:04.592955] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593003] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593029] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593055] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593063] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-23 >[2018-08-18 10:05:04.593064] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593100] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593097] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593122] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593133] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-24 >[2018-08-18 10:05:04.593155] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593200] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593201] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593234] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593241] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593249] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-25 >[2018-08-18 10:05:04.593249] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593282] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593310] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593311] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593328] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-26 >[2018-08-18 10:05:04.593346] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593347] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593374] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593380] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593385] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-27 >[2018-08-18 10:05:04.593467] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593470] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593499] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593501] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593527] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593545] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-29 >[2018-08-18 10:05:04.593591] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593618] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593630] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593654] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593665] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-30 >[2018-08-18 10:05:04.593666] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593703] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593736] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593748] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593757] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-31 >[2018-08-18 10:05:04.593769] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593780] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593784] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593820] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593838] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-32 >[2018-08-18 10:05:04.593853] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593863] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593859] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593898] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593895] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593919] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-33 >[2018-08-18 10:05:04.593922] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593921] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593936] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593938] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-28 >[2018-08-18 10:05:04.593946] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593978] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.593994] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-34 >[2018-08-18 10:05:04.594006] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.594000] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.594022] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.594058] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.594071] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006) from subvol vol0-replicate-35 >[2018-08-18 10:05:04.594084] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-28 cached_subvol null >[2018-08-18 10:05:04.594094] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006 >[2018-08-18 10:05:04.594125] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.594137] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.594150] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.594162] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.594173] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34030f40, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:04.594287] I [MSGID: 109066] [dht-rename.c:1610:dht_rename] 0-vol0-dht: renaming /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006_2ca8ee40a2ce11e88bc6246e9631871a (hash=vol0-replicate-2/cache=vol0-replicate-2) => /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006 (hash=vol0-replicate-28/cache=<nul>) >[2018-08-18 10:05:04.595107] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-28: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:04.597358] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.597377] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): picking the request for winding >[2018-08-18 10:05:04.597404] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/sjkzhdd_1/6d48364fa2cb11e880ef246e963180cc.t at vol0-replicate-21 >[2018-08-18 10:05:04.597607] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:04.597738] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/sjkzhdd_1/6d48364fa2cb11e880ef246e963180cc.t returned with op_ret 0 >[2018-08-18 10:05:04.598134] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7f3d0256d4e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7f3d0d44d631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7f3d02158820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.598153] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): picking the request for winding >[2018-08-18 10:05:04.598177] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-21: ebb1b4d9-763b-4b2e-9654-888177709dc4: generation now vs cached: 3, 3 >[2018-08-18 10:05:04.598312] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:04.598886] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.598914] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): picking the request for winding >[2018-08-18 10:05:04.598932] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:04.599171] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.599889] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:04.600088] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:04.600264] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.602385] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 1578284 is serviced using standard calloc() (0x7f3c240053d0) as it exceeds the maximum available buffer size >[2018-08-18 10:05:04.603473] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.603498] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): picking the request for winding >[2018-08-18 10:05:04.603513] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:04.603573] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.603586] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:04.604701] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.604728] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4): ordering.go is not set, hence not winding >[2018-08-18 10:05:04.607812] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:04.607973] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-10: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:04.608332] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 10:05:04.608487] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-29: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:04.608937] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:04.608990] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:04.609083] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-10: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:04.609099] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:04.609246] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.609268] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=1): picking the request for winding >[2018-08-18 10:05:04.609285] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 10:05:04.609341] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.609353] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:04.609401] D [socket.c:532:__socket_rwv] 0-vol0-client-63: would have passed zero length to read/write >[2018-08-18 10:05:04.609421] D [socket.c:532:__socket_rwv] 0-vol0-client-64: would have passed zero length to read/write >[2018-08-18 10:05:04.609423] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 10:05:04.609438] D [socket.c:532:__socket_rwv] 0-vol0-client-65: would have passed zero length to read/write >[2018-08-18 10:05:04.609450] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7f3c240053d0) allocated with standard calloc() >[2018-08-18 10:05:04.609484] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-29: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.609619] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.609699] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:04.609815] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:04.610006] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:04.610056] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=2): picking the request for winding >[2018-08-18 10:05:04.610072] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 10:05:04.610107] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.610129] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:04.610209] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.610443] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] )))processing queues >[2018-08-18 10:05:04.610494] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:04.610568] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.610594] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=d8824496-d2df-466a-818c-209a2e4f09aa, gen=0): picking the request for winding >[2018-08-18 10:05:04.610621] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:04.610631] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002 at vol0-replicate-10 >[2018-08-18 10:05:04.610650] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=3): picking the request for winding >[2018-08-18 10:05:04.610663] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 10:05:04.610694] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.610704] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:04.610731] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.610878] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=d8824496-d2df-466a-818c-209a2e4f09aa, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:04.610960] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002 returned with op_ret 0 >[2018-08-18 10:05:04.611016] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.611151] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:04.611258] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.611319] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=3): destroying request, removing from all queues >[2018-08-18 10:05:04.611364] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=4): picking the request for winding >[2018-08-18 10:05:04.611377] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=4): added req to liability queue. inode-generation-number=5 >[2018-08-18 10:05:04.611406] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.611415] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:04.611483] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.611498] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 10:05:04.611735] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.611756] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=d8824496-d2df-466a-818c-209a2e4f09aa, gen=0): picking the request for winding >[2018-08-18 10:05:04.611763] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.611784] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002 at vol0-replicate-10 >[2018-08-18 10:05:04.611872] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=4): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:04.612019] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=d8824496-d2df-466a-818c-209a2e4f09aa, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:04.612084] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002 returned with op_ret 0 >[2018-08-18 10:05:04.612091] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=4): destroying request, removing from all queues >[2018-08-18 10:05:04.612153] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=5): picking the request for winding >[2018-08-18 10:05:04.612296] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.612311] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.612378] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=ebb1b4d9-763b-4b2e-9654-888177709dc4, gen=5): destroying request, removing from all queues >[2018-08-18 10:05:04.612510] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 10:05:04.612508] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:04.612729] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.612749] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=d8824496-d2df-466a-818c-209a2e4f09aa, gen=0): picking the request for winding >[2018-08-18 10:05:04.612779] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002 at vol0-replicate-10 >[2018-08-18 10:05:04.613008] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=d8824496-d2df-466a-818c-209a2e4f09aa, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:04.613069] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002 returned with op_ret 0 >[2018-08-18 10:05:04.613293] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.613486] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 10:05:04.613699] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.613718] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=d8824496-d2df-466a-818c-209a2e4f09aa, gen=0): picking the request for winding >[2018-08-18 10:05:04.613745] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002 at vol0-replicate-10 >[2018-08-18 10:05:04.613959] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=d8824496-d2df-466a-818c-209a2e4f09aa, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:04.614048] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0002 returned with op_ret 0 >[2018-08-18 10:05:04.614307] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.614489] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 10:05:04.618206] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 10:05:04.618328] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-2: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:04.618654] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 10:05:04.618777] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-28: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:04.619330] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 10:05:04.619451] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-2: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:04.619776] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 10:05:04.619838] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-28: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.620045] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 10:05:04.620545] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.620564] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=a0aa13d4-bc23-45a2-9478-e6420c92b07e, gen=0): picking the request for winding >[2018-08-18 10:05:04.620586] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006 at vol0-replicate-2 >[2018-08-18 10:05:04.620775] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=a0aa13d4-bc23-45a2-9478-e6420c92b07e, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:04.620882] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006 returned with op_ret 0 >[2018-08-18 10:05:04.621150] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-2: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.621372] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 10:05:04.621664] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.621681] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=a0aa13d4-bc23-45a2-9478-e6420c92b07e, gen=0): picking the request for winding >[2018-08-18 10:05:04.621702] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006 at vol0-replicate-2 >[2018-08-18 10:05:04.621900] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=a0aa13d4-bc23-45a2-9478-e6420c92b07e, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:04.621995] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006 returned with op_ret 0 >[2018-08-18 10:05:04.622224] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-2: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.622426] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 10:05:04.622621] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.622638] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=a0aa13d4-bc23-45a2-9478-e6420c92b07e, gen=0): picking the request for winding >[2018-08-18 10:05:04.622658] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006 at vol0-replicate-2 >[2018-08-18 10:05:04.622845] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=a0aa13d4-bc23-45a2-9478-e6420c92b07e, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:04.622955] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006 returned with op_ret 0 >[2018-08-18 10:05:04.623164] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-2: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.623344] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 10:05:04.623542] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:04.623558] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=a0aa13d4-bc23-45a2-9478-e6420c92b07e, gen=0): picking the request for winding >[2018-08-18 10:05:04.623578] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006 at vol0-replicate-2 >[2018-08-18 10:05:04.623767] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=a0aa13d4-bc23-45a2-9478-e6420c92b07e, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:04.623860] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/892a0f35ba7f3257a399289abeb8db59_171632295487241529.m/0006 returned with op_ret 0 >[2018-08-18 10:05:04.624047] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-2: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:04.624240] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 10:05:05.097305] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m on vol0-replicate-5 >[2018-08-18 10:05:05.098161] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m with op_ret 0 >[2018-08-18 10:05:05.103490] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a on vol0-replicate-1 >[2018-08-18 10:05:05.103797] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.103867] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.103868] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.103934] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.103960] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a with op_ret -1 [No such file or directory] >[2018-08-18 10:05:05.103983] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a missing on subvol vol0-replicate-1 >[2018-08-18 10:05:05.104003] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:05.104360] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.104385] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.104428] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.104464] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.104480] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-0 >[2018-08-18 10:05:05.104499] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.104504] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.104587] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.104610] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.104649] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.104691] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.104719] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-2 >[2018-08-18 10:05:05.104730] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.104720] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.104768] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.104803] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.104820] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-3 >[2018-08-18 10:05:05.104857] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.104882] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.104859] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.104922] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.104936] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-4 >[2018-08-18 10:05:05.104949] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.104959] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.104960] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105008] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105022] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-5 >[2018-08-18 10:05:05.105025] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105028] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105081] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105097] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105112] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105109] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105129] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-6 >[2018-08-18 10:05:05.105233] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105253] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105255] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105309] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105322] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105311] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105315] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105350] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-8 >[2018-08-18 10:05:05.105310] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105361] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105372] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105379] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-7 >[2018-08-18 10:05:05.105384] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-9 >[2018-08-18 10:05:05.105390] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105421] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105480] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105488] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105502] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105518] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105519] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-10 >[2018-08-18 10:05:05.105560] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105599] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105605] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105612] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105622] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-11 >[2018-08-18 10:05:05.105649] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105668] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105691] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-12 >[2018-08-18 10:05:05.105700] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105732] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105785] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105828] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105821] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105822] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105867] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-13 >[2018-08-18 10:05:05.105893] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105931] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105934] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105931] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105972] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.105974] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-14 >[2018-08-18 10:05:05.106020] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106028] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106059] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106055] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-15 >[2018-08-18 10:05:05.106090] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106129] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106142] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106148] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-16 >[2018-08-18 10:05:05.106143] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106153] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106203] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106208] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106222] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-17 >[2018-08-18 10:05:05.106241] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106244] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106273] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106293] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-18 >[2018-08-18 10:05:05.106309] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106317] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106326] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106349] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106364] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-19 >[2018-08-18 10:05:05.106385] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106416] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106484] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106513] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106516] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106526] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-20 >[2018-08-18 10:05:05.106550] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106589] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106602] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106610] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-21 >[2018-08-18 10:05:05.106612] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106630] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106657] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106685] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106701] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-22 >[2018-08-18 10:05:05.106702] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106701] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106736] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106756] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106768] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-23 >[2018-08-18 10:05:05.106808] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106836] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106839] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106862] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106878] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-24 >[2018-08-18 10:05:05.106905] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106924] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106941] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106977] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.106991] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-25 >[2018-08-18 10:05:05.107024] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107064] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107070] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107114] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107128] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-26 >[2018-08-18 10:05:05.107138] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107175] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107206] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107210] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107233] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107256] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107260] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107275] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-27 >[2018-08-18 10:05:05.107289] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107313] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-28 >[2018-08-18 10:05:05.107316] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107329] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107350] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107371] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107383] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-29 >[2018-08-18 10:05:05.107404] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107437] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107472] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107493] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107505] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-30 >[2018-08-18 10:05:05.107513] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107556] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107562] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107602] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107615] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-31 >[2018-08-18 10:05:05.107634] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107643] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107690] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107715] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107729] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-32 >[2018-08-18 10:05:05.107744] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107777] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107793] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107815] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107826] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-33 >[2018-08-18 10:05:05.107856] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107869] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107876] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107893] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107919] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107920] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107942] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-1 >[2018-08-18 10:05:05.107949] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-34 >[2018-08-18 10:05:05.107984] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107985] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.107986] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.108027] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.108067] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a) from subvol vol0-replicate-35 >[2018-08-18 10:05:05.108098] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-1 cached_subvol null >[2018-08-18 10:05:05.108111] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a >[2018-08-18 10:05:05.108123] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.108135] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.108150] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.108162] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.108175] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.108502] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-0': avail_percent is: 96.00 and avail_space is: 7683154767872 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.108531] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-1': avail_percent is: 95.00 and avail_space is: 7649958682624 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.108581] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-2': avail_percent is: 96.00 and avail_space is: 7682652368896 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.108619] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-3': avail_percent is: 96.00 and avail_space is: 7679958376448 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.108671] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-4': avail_percent is: 96.00 and avail_space is: 7680306479104 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.108717] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-5': avail_percent is: 95.00 and avail_space is: 7674803892224 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.108745] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-6': avail_percent is: 94.00 and avail_space is: 7572583333888 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.108778] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-7': avail_percent is: 95.00 and avail_space is: 7602560430080 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.108851] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-8': avail_percent is: 95.00 and avail_space is: 7674537177088 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.108892] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-9': avail_percent is: 95.00 and avail_space is: 7675663728640 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.108942] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-10': avail_percent is: 94.00 and avail_space is: 7587969724416 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.108971] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-11': avail_percent is: 95.00 and avail_space is: 7666756444160 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109024] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-12': avail_percent is: 95.00 and avail_space is: 7662852456448 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109073] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-13': avail_percent is: 95.00 and avail_space is: 7672171716608 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109096] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-14': avail_percent is: 95.00 and avail_space is: 7671006871552 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109147] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-15': avail_percent is: 95.00 and avail_space is: 7657355005952 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109203] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-16': avail_percent is: 96.00 and avail_space is: 7681130377216 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109238] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-17': avail_percent is: 95.00 and avail_space is: 7670385815552 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109315] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-18': avail_percent is: 95.00 and avail_space is: 7620654678016 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109351] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-19': avail_percent is: 95.00 and avail_space is: 7661022814208 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109387] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-20': avail_percent is: 95.00 and avail_space is: 7611642155008 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109416] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-21': avail_percent is: 95.00 and avail_space is: 7663950176256 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109470] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-22': avail_percent is: 95.00 and avail_space is: 7668892258304 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109515] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-23': avail_percent is: 95.00 and avail_space is: 7670234238976 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109554] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-24': avail_percent is: 95.00 and avail_space is: 7665744367616 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109586] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-25': avail_percent is: 95.00 and avail_space is: 7650961227776 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109609] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-26': avail_percent is: 95.00 and avail_space is: 7659003916288 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109652] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-27': avail_percent is: 95.00 and avail_space is: 7652756992000 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109703] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-28': avail_percent is: 95.00 and avail_space is: 7652597972992 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109734] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-29': avail_percent is: 95.00 and avail_space is: 7639883448320 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109782] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-30': avail_percent is: 95.00 and avail_space is: 7677999882240 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109824] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-31': avail_percent is: 95.00 and avail_space is: 7678396768256 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109863] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-32': avail_percent is: 95.00 and avail_space is: 7664158932992 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109897] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-33': avail_percent is: 95.00 and avail_space is: 7679374594048 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.109919] D [MSGID: 0] [dht-common.c:6923:dht_create_wind_to_avail_subvol] 0-vol0-dht: creating /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a on vol0-replicate-1 >[2018-08-18 10:05:05.109995] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-34': avail_percent is: 95.00 and avail_space is: 7668156858368 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.110044] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-35': avail_percent is: 95.00 and avail_space is: 7666712301568 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.110117] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-1: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.142594] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-1 >[2018-08-18 10:05:05.142663] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 10:05:05.247278] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-22 >[2018-08-18 10:05:05.247656] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.247677] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.247706] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.247753] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.247772] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m with op_ret -1 [No such file or directory] >[2018-08-18 10:05:05.247789] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m missing on subvol vol0-replicate-22 >[2018-08-18 10:05:05.247813] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:05.248172] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248235] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248237] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248297] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248318] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-0 >[2018-08-18 10:05:05.248327] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248329] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248374] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248392] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248393] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248393] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248445] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-1 >[2018-08-18 10:05:05.248448] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248478] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248490] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-2 >[2018-08-18 10:05:05.248492] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248524] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248525] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248597] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248616] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248622] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248640] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248641] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-3 >[2018-08-18 10:05:05.248657] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248667] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-4 >[2018-08-18 10:05:05.248725] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248765] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248787] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248831] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248850] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-5 >[2018-08-18 10:05:05.248872] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248901] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248909] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248967] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.248996] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-6 >[2018-08-18 10:05:05.249007] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249044] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249051] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249101] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249103] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249132] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-7 >[2018-08-18 10:05:05.249136] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249157] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249200] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249223] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-8 >[2018-08-18 10:05:05.249213] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249253] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249255] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249273] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249292] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249306] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-9 >[2018-08-18 10:05:05.249298] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249315] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249343] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249354] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-10 >[2018-08-18 10:05:05.249394] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249437] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249445] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249475] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249492] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-11 >[2018-08-18 10:05:05.249509] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249525] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249565] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249612] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249630] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249627] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249629] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-12 >[2018-08-18 10:05:05.249667] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249680] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249690] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-13 >[2018-08-18 10:05:05.249703] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249735] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249804] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249833] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249846] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-14 >[2018-08-18 10:05:05.249851] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249872] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249898] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249937] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.249955] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-15 >[2018-08-18 10:05:05.249972] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250025] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250019] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250081] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250080] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250106] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-16 >[2018-08-18 10:05:05.250132] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250139] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250157] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250188] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250208] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250218] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-18 >[2018-08-18 10:05:05.250268] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250273] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250291] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250321] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250338] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-19 >[2018-08-18 10:05:05.250353] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250410] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250420] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250452] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250463] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-20 >[2018-08-18 10:05:05.250458] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250459] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250497] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250512] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-17 >[2018-08-18 10:05:05.250517] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250581] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250579] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250607] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250637] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250670] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-21 >[2018-08-18 10:05:05.250671] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250705] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250725] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250758] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250773] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-23 >[2018-08-18 10:05:05.250825] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250832] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250835] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250865] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250877] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-24 >[2018-08-18 10:05:05.250901] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250933] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250924] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250982] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.250987] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251009] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-25 >[2018-08-18 10:05:05.251015] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251010] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251083] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251110] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-26 >[2018-08-18 10:05:05.251125] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251145] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251187] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251226] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251243] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-27 >[2018-08-18 10:05:05.251247] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251288] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251308] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251327] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251337] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-28 >[2018-08-18 10:05:05.251392] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251408] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251472] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251499] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251519] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-29 >[2018-08-18 10:05:05.251545] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251550] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251584] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251614] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251628] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-30 >[2018-08-18 10:05:05.251640] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251665] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251674] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251695] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251699] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251715] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251734] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-31 >[2018-08-18 10:05:05.251752] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251801] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251817] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-32 >[2018-08-18 10:05:05.251841] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251871] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251879] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251905] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251920] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-33 >[2018-08-18 10:05:05.251924] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251952] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251958] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251982] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-22 >[2018-08-18 10:05:05.251978] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.251991] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.252026] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.252053] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-34 >[2018-08-18 10:05:05.252070] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.252072] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.252125] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.252153] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.252166] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-35 >[2018-08-18 10:05:05.252181] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-22 cached_subvol null >[2018-08-18 10:05:05.252193] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m >[2018-08-18 10:05:05.252204] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.252217] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.252233] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.252247] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.252261] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.253111] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0 returned with op_ret 0 >[2018-08-18 10:05:05.256887] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0 returned with op_ret 0" repeated 35 times between [2018-08-18 10:05:05.253111] and [2018-08-18 10:05:05.256504] >[2018-08-18 10:05:05.256886] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-22 >[2018-08-18 10:05:05.257181] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.257218] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.257227] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.257279] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.257298] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m with op_ret -1 [No such file or directory] >[2018-08-18 10:05:05.257313] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m missing on subvol vol0-replicate-22 >[2018-08-18 10:05:05.257336] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:05.257659] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.257671] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.257683] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.257715] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.257731] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-0 >[2018-08-18 10:05:05.257792] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.257792] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.257817] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.257856] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.257882] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.257886] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.257898] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-1 >[2018-08-18 10:05:05.257900] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.257893] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.257941] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.257953] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.257985] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-2 >[2018-08-18 10:05:05.258050] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258052] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258049] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258105] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258112] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258130] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258144] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258150] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-3 >[2018-08-18 10:05:05.258163] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258172] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-4 >[2018-08-18 10:05:05.258205] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258242] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258247] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258243] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258264] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-5 >[2018-08-18 10:05:05.258283] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258315] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258334] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-6 >[2018-08-18 10:05:05.258386] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258433] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258443] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258467] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258477] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-7 >[2018-08-18 10:05:05.258563] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258585] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258594] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258636] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258658] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-8 >[2018-08-18 10:05:05.258659] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258649] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258713] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258733] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258733] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258745] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-9 >[2018-08-18 10:05:05.258746] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258834] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258843] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258868] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258869] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258914] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258928] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-11 >[2018-08-18 10:05:05.258931] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258934] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258950] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-10 >[2018-08-18 10:05:05.258979] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258982] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.258982] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259019] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259080] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259084] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259087] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259121] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259126] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259128] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-12 >[2018-08-18 10:05:05.259144] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-13 >[2018-08-18 10:05:05.259184] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259195] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259227] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259238] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259230] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259270] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-14 >[2018-08-18 10:05:05.259274] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259300] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-15 >[2018-08-18 10:05:05.259302] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259317] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259350] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259406] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259415] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259418] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259452] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-16 >[2018-08-18 10:05:05.259487] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259498] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259527] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259546] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259551] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259581] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259546] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-17 >[2018-08-18 10:05:05.259610] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259610] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259644] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259656] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-18 >[2018-08-18 10:05:05.259671] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259683] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-19 >[2018-08-18 10:05:05.259701] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259746] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259768] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259787] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259792] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259806] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-20 >[2018-08-18 10:05:05.259821] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259849] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259873] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259886] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-21 >[2018-08-18 10:05:05.259901] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259925] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.259984] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260006] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260017] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260050] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260058] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-23 >[2018-08-18 10:05:05.260102] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260103] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260134] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260155] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260165] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-24 >[2018-08-18 10:05:05.260186] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260233] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260234] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260305] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260313] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260310] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260349] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-25 >[2018-08-18 10:05:05.260373] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260386] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260404] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260417] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260426] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-26 >[2018-08-18 10:05:05.260444] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260471] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260478] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260481] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260499] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260509] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-27 >[2018-08-18 10:05:05.260524] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260543] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260544] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-28 >[2018-08-18 10:05:05.260571] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260601] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260626] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260645] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-29 >[2018-08-18 10:05:05.260666] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260710] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260744] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260772] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260781] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260785] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-30 >[2018-08-18 10:05:05.260817] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260865] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260884] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260895] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260918] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-31 >[2018-08-18 10:05:05.260930] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260925] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260989] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260986] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260991] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.260994] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.261040] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-32 >[2018-08-18 10:05:05.261054] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.261071] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-33 >[2018-08-18 10:05:05.261082] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.261099] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.261090] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.261127] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.261119] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.261152] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.261152] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-22 >[2018-08-18 10:05:05.261165] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-34 >[2018-08-18 10:05:05.261224] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.261228] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.261239] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.261311] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.261343] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m) from subvol vol0-replicate-35 >[2018-08-18 10:05:05.261371] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-22 cached_subvol null >[2018-08-18 10:05:05.261392] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m >[2018-08-18 10:05:05.261412] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.261435] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.261460] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.261483] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.261507] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.262097] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-22: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.267532] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7f3c34057ad0) as it exceeds the maximum available buffer size >[2018-08-18 10:05:05.268796] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.268831] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=30e78aeb-d190-4a89-bf3b-03616bdf8153, gen=0): picking the request for winding >[2018-08-18 10:05:05.268857] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=30e78aeb-d190-4a89-bf3b-03616bdf8153, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:05.269178] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-1: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.277253] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS on vol0-replicate-24 >[2018-08-18 10:05:05.277810] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49212: ping timer event already removed >[2018-08-18 10:05:05.278162] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49212: ping timer event already removed >[2018-08-18 10:05:05.278354] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS on vol0-replicate-24 >[2018-08-18 10:05:05.278531] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49212: ping timer event already removed >[2018-08-18 10:05:05.278784] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS with op_ret 0 >[2018-08-18 10:05:05.279026] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49188: ping timer event already removed >[2018-08-18 10:05:05.279198] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49188: ping timer event already removed >[2018-08-18 10:05:05.279369] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49188: ping timer event already removed >[2018-08-18 10:05:05.279571] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49189: ping timer event already removed >[2018-08-18 10:05:05.279724] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49189: ping timer event already removed >[2018-08-18 10:05:05.279900] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49189: ping timer event already removed >[2018-08-18 10:05:05.279884] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS with op_ret 0 >[2018-08-18 10:05:05.280089] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49190: ping timer event already removed >[2018-08-18 10:05:05.280244] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49190: ping timer event already removed >[2018-08-18 10:05:05.280397] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49190: ping timer event already removed >[2018-08-18 10:05:05.280598] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49191: ping timer event already removed >[2018-08-18 10:05:05.280752] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49191: ping timer event already removed >[2018-08-18 10:05:05.280902] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49191: ping timer event already removed >[2018-08-18 10:05:05.281111] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49192: ping timer event already removed >[2018-08-18 10:05:05.281288] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49192: ping timer event already removed >[2018-08-18 10:05:05.281454] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49192: ping timer event already removed >[2018-08-18 10:05:05.281661] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49193: ping timer event already removed >[2018-08-18 10:05:05.281923] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49193: ping timer event already removed >[2018-08-18 10:05:05.282158] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49193: ping timer event already removed >[2018-08-18 10:05:05.282365] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49194: ping timer event already removed >[2018-08-18 10:05:05.282520] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49194: ping timer event already removed >[2018-08-18 10:05:05.282664] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49194: ping timer event already removed >[2018-08-18 10:05:05.282843] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49195: ping timer event already removed >[2018-08-18 10:05:05.282998] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49195: ping timer event already removed >[2018-08-18 10:05:05.283156] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49195: ping timer event already removed >[2018-08-18 10:05:05.283334] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49196: ping timer event already removed >[2018-08-18 10:05:05.283480] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49196: ping timer event already removed >[2018-08-18 10:05:05.283522] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-0: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.283626] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49196: ping timer event already removed >[2018-08-18 10:05:05.283654] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-3: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.283682] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-4: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.283717] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-2: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.283731] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-5: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.283766] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-6: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.283798] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49197: ping timer event already removed >[2018-08-18 10:05:05.283856] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-7: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.283919] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-8: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.283958] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49197: ping timer event already removed >[2018-08-18 10:05:05.283970] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-9: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.284136] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-10: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.284150] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49197: ping timer event already removed >[2018-08-18 10:05:05.284207] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-11: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.284291] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-13: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.284304] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-12: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.284334] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 10:05:05.284352] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-1: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.284367] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49198: ping timer event already removed >[2018-08-18 10:05:05.284395] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 10:05:05.284497] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-14: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.284554] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-15: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.284593] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-16: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.284625] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-17: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.284744] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49198: ping timer event already removed >[2018-08-18 10:05:05.284780] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-19: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.284807] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-18: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.284869] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-20: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.284930] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-21: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.284955] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:05.284996] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-23: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.285072] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-24: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.285134] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49198: ping timer event already removed >[2018-08-18 10:05:05.285163] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-25: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.285219] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-26: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.285261] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:05.285346] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-27: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.285416] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-28: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.285467] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-29: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.285545] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-26: lk op is for a transaction >[2018-08-18 10:05:05.285608] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49199: ping timer event already removed >[2018-08-18 10:05:05.285630] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-31: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.285649] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-30: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.285663] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-27: lk op is for a transaction >[2018-08-18 10:05:05.285761] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-32: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.285767] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 10:05:05.285789] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-22: lk op is for a transaction >[2018-08-18 10:05:05.285835] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:05.285853] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-33: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.285874] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-34: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.285935] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-35: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.286075] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49199: ping timer event already removed >[2018-08-18 10:05:05.286144] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 10:05:05.286308] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-30: lk op is for a transaction >[2018-08-18 10:05:05.286401] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49199: ping timer event already removed >[2018-08-18 10:05:05.286770] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49200: ping timer event already removed >[2018-08-18 10:05:05.286872] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=30e78aeb-d190-4a89-bf3b-03616bdf8153, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:05.286905] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt on vol0-replicate-34 >[2018-08-18 10:05:05.287011] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=30e78aeb-d190-4a89-bf3b-03616bdf8153, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.287062] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49200: ping timer event already removed >[2018-08-18 10:05:05.287162] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.287203] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt with op_ret 0 >[2018-08-18 10:05:05.287337] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49200: ping timer event already removed >[2018-08-18 10:05:05.287690] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49201: ping timer event already removed >[2018-08-18 10:05:05.287968] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49201: ping timer event already removed >[2018-08-18 10:05:05.288275] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49201: ping timer event already removed >[2018-08-18 10:05:05.288624] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49202: ping timer event already removed >[2018-08-18 10:05:05.288935] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49202: ping timer event already removed >[2018-08-18 10:05:05.289257] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49202: ping timer event already removed >[2018-08-18 10:05:05.289807] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49204: ping timer event already removed >[2018-08-18 10:05:05.290129] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49204: ping timer event already removed >[2018-08-18 10:05:05.290409] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49204: ping timer event already removed >[2018-08-18 10:05:05.290732] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49205: ping timer event already removed >[2018-08-18 10:05:05.291002] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49205: ping timer event already removed >[2018-08-18 10:05:05.291263] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx on vol0-replicate-20 >[2018-08-18 10:05:05.291296] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49205: ping timer event already removed >[2018-08-18 10:05:05.291586] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx with op_ret 0 >[2018-08-18 10:05:05.291599] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49206: ping timer event already removed >[2018-08-18 10:05:05.291862] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49206: ping timer event already removed >[2018-08-18 10:05:05.292199] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49206: ping timer event already removed >[2018-08-18 10:05:05.292508] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49207: ping timer event already removed >[2018-08-18 10:05:05.292795] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49207: ping timer event already removed >[2018-08-18 10:05:05.293074] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49207: ping timer event already removed >[2018-08-18 10:05:05.293432] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49208: ping timer event already removed >[2018-08-18 10:05:05.293702] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49208: ping timer event already removed >[2018-08-18 10:05:05.293978] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49208: ping timer event already removed >[2018-08-18 10:05:05.294380] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49209: ping timer event already removed >[2018-08-18 10:05:05.294599] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0 on vol0-replicate-23 >[2018-08-18 10:05:05.294651] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49209: ping timer event already removed >[2018-08-18 10:05:05.294910] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49209: ping timer event already removed >[2018-08-18 10:05:05.294945] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0 with op_ret 0 >[2018-08-18 10:05:05.295288] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49210: ping timer event already removed >[2018-08-18 10:05:05.295618] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49210: ping timer event already removed >[2018-08-18 10:05:05.295901] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49210: ping timer event already removed >[2018-08-18 10:05:05.296264] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49211: ping timer event already removed >[2018-08-18 10:05:05.296513] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49211: ping timer event already removed >[2018-08-18 10:05:05.296772] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49211: ping timer event already removed >[2018-08-18 10:05:05.297235] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49213: ping timer event already removed >[2018-08-18 10:05:05.297572] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49213: ping timer event already removed >[2018-08-18 10:05:05.297829] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49213: ping timer event already removed >[2018-08-18 10:05:05.298131] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49214: ping timer event already removed >[2018-08-18 10:05:05.298193] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-22 >[2018-08-18 10:05:05.298430] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49214: ping timer event already removed >[2018-08-18 10:05:05.298486] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m with op_ret 0 >[2018-08-18 10:05:05.298671] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49214: ping timer event already removed >[2018-08-18 10:05:05.298774] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.298795] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.298807] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.298863] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.298888] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.298863] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.298885] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.298927] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.298952] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-1 returned error [No such file or directory] >[2018-08-18 10:05:05.298983] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.298983] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299045] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299066] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-3 returned error [No such file or directory] >[2018-08-18 10:05:05.299076] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299076] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299097] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49215: ping timer event already removed >[2018-08-18 10:05:05.299127] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299173] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299165] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299214] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299246] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-5 returned error [No such file or directory] >[2018-08-18 10:05:05.299316] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299332] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299352] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299395] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299417] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-8 returned error [No such file or directory] >[2018-08-18 10:05:05.299419] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-22 >[2018-08-18 10:05:05.299485] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299505] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c30001f40, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299516] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49215: ping timer event already removed >[2018-08-18 10:05:05.299530] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c001f40, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299545] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c30001f40, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299598] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299594] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299628] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299664] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299677] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-10 returned error [No such file or directory] >[2018-08-18 10:05:05.299682] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299682] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299694] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299727] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299729] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299734] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299749] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299758] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-11 returned error [No such file or directory] >[2018-08-18 10:05:05.299779] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299794] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-12 returned error [No such file or directory] >[2018-08-18 10:05:05.299809] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299802] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299818] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49215: ping timer event already removed >[2018-08-18 10:05:05.299858] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m with op_ret 0 >[2018-08-18 10:05:05.299947] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.299969] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300022] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300140] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49216: ping timer event already removed >[2018-08-18 10:05:05.300171] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300271] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300297] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300330] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300352] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300374] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300384] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-1 returned error [No such file or directory] >[2018-08-18 10:05:05.300407] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49216: ping timer event already removed >[2018-08-18 10:05:05.300423] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300487] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300494] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300018] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300533] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300561] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300562] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300581] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300596] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300594] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-16 returned error [No such file or directory] >[2018-08-18 10:05:05.300611] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300641] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300656] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-17 returned error [No such file or directory] >[2018-08-18 10:05:05.300660] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49216: ping timer event already removed >[2018-08-18 10:05:05.300668] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300725] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300743] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300754] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-18 returned error [No such file or directory] >[2018-08-18 10:05:05.300777] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300798] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300810] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300842] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300858] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-19 returned error [No such file or directory] >[2018-08-18 10:05:05.300912] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-0: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.300934] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2c001f40, vol0-client-0 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.300940] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300952] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300973] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-1: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.300991] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.300993] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-2: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.300999] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c1c001f40, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301019] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301008] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49217: ping timer event already removed >[2018-08-18 10:05:05.301045] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-3 returned error [No such file or directory] >[2018-08-18 10:05:05.301022] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2c001f40, vol0-client-2 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.301049] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2c001f40, vol0-client-1 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.301021] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c00bcb0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301102] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c1c001f40, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301106] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c00bcb0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301108] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c24001f40, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301186] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301186] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301195] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301210] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301232] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301234] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301238] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301251] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-6: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.301263] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-7: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.301287] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c30001f40, vol0-client-6 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.301267] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301296] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c30001f40, vol0-client-7 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.301312] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-24 returned error [No such file or directory] >[2018-08-18 10:05:05.301329] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301332] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301338] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-8: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.301315] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c24001f40, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301360] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c30001f40, vol0-client-8 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.301353] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301354] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c28001f40, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301403] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301403] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49217: ping timer event already removed >[2018-08-18 10:05:05.301426] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301428] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c28001f40, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301454] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-5 returned error [No such file or directory] >[2018-08-18 10:05:05.301557] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 10:05:05.301665] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301674] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301677] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301719] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301735] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-8 returned error [No such file or directory] >[2018-08-18 10:05:05.301782] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2400bcb0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301792] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49217: ping timer event already removed >[2018-08-18 10:05:05.301807] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301837] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2400bcb0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301874] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c009700, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301881] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301884] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-0: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.301903] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301916] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-1: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.301908] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-27: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.301905] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301919] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c24001f40, vol0-client-0 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.301918] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-28: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.301940] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c24001f40, vol0-client-1 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.301944] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301945] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-2: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.301964] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-10 returned error [No such file or directory] >[2018-08-18 10:05:05.301939] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3000bcb0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301970] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c24001f40, vol0-client-2 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.301939] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c24001f40, vol0-client-27 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.302004] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.301888] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c009700, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302012] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302003] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c24001f40, vol0-client-28 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.302041] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-29: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.302056] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302061] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c24001f40, vol0-client-29 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.302072] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302080] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302080] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302095] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-11 returned error [No such file or directory] >[2018-08-18 10:05:05.302093] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302084] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3000bcb0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302114] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302129] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302111] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302148] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-31 returned error [No such file or directory] >[2018-08-18 10:05:05.302135] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302163] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 10:05:05.302185] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302187] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302202] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-12 returned error [No such file or directory] >[2018-08-18 10:05:05.302194] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302195] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302242] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302253] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302245] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302260] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-33 returned error [No such file or directory] >[2018-08-18 10:05:05.302272] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302296] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302322] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302318] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 10:05:05.302317] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-6: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.302340] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302336] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-34 returned error [No such file or directory] >[2018-08-18 10:05:05.302352] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c28001f40, vol0-client-6 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.302374] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-8: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.302387] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302387] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-13: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.302404] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302376] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-12: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.302415] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2400bcb0, vol0-client-13 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.302408] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c28001f40, vol0-client-8 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.302424] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302437] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-35 returned error [No such file or directory] >[2018-08-18 10:05:05.302431] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2400bcb0, vol0-client-12 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.302440] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-12: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.302439] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-7: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.302454] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-13: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.302463] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c28001f40, vol0-client-7 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.302463] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2c00bcb0, vol0-client-12 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.302472] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2c00bcb0, vol0-client-13 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.302479] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-14: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.302487] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-14: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.302496] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2400bcb0, vol0-client-14 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.302503] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2c00bcb0, vol0-client-14 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.302564] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302584] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302582] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49218: ping timer event already removed >[2018-08-18 10:05:05.302632] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302696] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302712] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-39: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.302722] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c1c001f40, vol0-client-39 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.302712] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302724] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-16 returned error [No such file or directory] >[2018-08-18 10:05:05.302761] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-41: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.302763] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-40: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.302775] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c1c001f40, vol0-client-41 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.302794] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c1c001f40, vol0-client-40 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.302826] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-60: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.302833] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302836] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.302842] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-61: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.302846] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-62: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.302872] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2c009700, vol0-client-61 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.302842] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2c009700, vol0-client-60 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.302892] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2c009700, vol0-client-62 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.302938] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49218: ping timer event already removed >[2018-08-18 10:05:05.303006] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-69: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.303011] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-70: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.303015] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.303029] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c3000bcb0, vol0-client-69 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.303028] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c3000bcb0, vol0-client-70 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.303025] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-71: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.303065] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.303076] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c3000bcb0, vol0-client-71 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.303091] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.303113] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.303320] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c20001f40, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.303338] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49218: ping timer event already removed >[2018-08-18 10:05:05.303358] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c24014b80, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.303394] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2800bcb0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.303413] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c24014b80, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.303529] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.303549] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.303677] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.303685] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.303700] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.303739] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.303740] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 10:05:05.303756] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-24 returned error [No such file or directory] >[2018-08-18 10:05:05.303915] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49219: ping timer event already removed >[2018-08-18 10:05:05.303957] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c28013860, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.304002] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c14001f40, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.304012] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-39: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.304041] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c20001f40, vol0-client-39 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.304046] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c28013860, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.304087] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c14001f40, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.304280] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49219: ping timer event already removed >[2018-08-18 10:05:05.304360] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-54: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.304368] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-56: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.304375] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-55: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.304387] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c24014b80, vol0-client-54 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.304393] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c24014b80, vol0-client-56 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.304402] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c24014b80, vol0-client-55 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.304493] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.304520] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.304576] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.304576] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 10:05:05.304635] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-51: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.304635] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.304637] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-52: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.304691] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2800bcb0, vol0-client-51 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.304694] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-53: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.304724] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2800bcb0, vol0-client-52 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.304730] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-31 returned error [No such file or directory] >[2018-08-18 10:05:05.304751] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2800bcb0, vol0-client-53 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.304795] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-57: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.304805] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.304824] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.304819] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c14001f40, vol0-client-57 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.304818] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-58: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.304863] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49219: ping timer event already removed >[2018-08-18 10:05:05.304878] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c14001f40, vol0-client-58 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.304879] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-59: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.304889] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c24009700, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.304904] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.304888] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c24009700, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.304924] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c14001f40, vol0-client-59 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.304946] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.304969] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.304978] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-60: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.304997] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.305006] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-61: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.305023] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c28013860, vol0-client-60 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.305018] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-34 returned error [No such file or directory] >[2018-08-18 10:05:05.304997] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.305013] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-62: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.305074] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.305101] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c28013860, vol0-client-62 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.305105] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.305106] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c28013860, vol0-client-61 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.305141] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.305169] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-35 returned error [No such file or directory] >[2018-08-18 10:05:05.305241] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49220: ping timer event already removed >[2018-08-18 10:05:05.305530] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-69: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.305539] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-70: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.305567] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c24009700, vol0-client-69 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.305569] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-71: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.305580] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c24009700, vol0-client-70 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.305577] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49220: ping timer event already removed >[2018-08-18 10:05:05.305599] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c24009700, vol0-client-71 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.305608] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2000aea0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.305639] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2000aea0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.305920] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49220: ping timer event already removed >[2018-08-18 10:05:05.306068] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-99: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.306079] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-100: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.306096] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2000aea0, vol0-client-99 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.306103] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2000aea0, vol0-client-100 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.306104] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-101: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.306128] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2000aea0, vol0-client-101 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.306343] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49221: ping timer event already removed >[2018-08-18 10:05:05.306626] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49221: ping timer event already removed >[2018-08-18 10:05:05.306895] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49221: ping timer event already removed >[2018-08-18 10:05:05.307234] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49222: ping timer event already removed >[2018-08-18 10:05:05.307493] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49222: ping timer event already removed >[2018-08-18 10:05:05.307684] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 10:05:05.307727] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49222: ping timer event already removed >[2018-08-18 10:05:05.308071] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49223: ping timer event already removed >[2018-08-18 10:05:05.308208] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:05.308241] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 10:05:05.308340] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49223: ping timer event already removed >[2018-08-18 10:05:05.308662] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49223: ping timer event already removed >[2018-08-18 10:05:05.309044] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt on vol0-replicate-34 >[2018-08-18 10:05:05.309227] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 10:05:05.309422] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt with op_ret 0 >[2018-08-18 10:05:05.309756] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 10:05:05.310462] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 10:05:05.310728] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 10:05:05.310898] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-23: lk op is for a transaction >[2018-08-18 10:05:05.311328] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 10:05:05.311683] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 10:05:05.313384] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:05.313435] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 10:05:05.314686] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 10:05:05.314866] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 10:05:05.315412] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 10:05:05.315928] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 10:05:05.316853] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx on vol0-replicate-20 >[2018-08-18 10:05:05.317221] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx with op_ret 0 >[2018-08-18 10:05:05.317382] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 10:05:05.318006] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 10:05:05.321906] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-20: lk op is for a transaction >[2018-08-18 10:05:05.322266] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0 on vol0-replicate-23 >[2018-08-18 10:05:05.322706] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0 with op_ret 0 >[2018-08-18 10:05:05.324268] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-vol0-dht: Found anomalies in /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m (gfid = b31240fd-5c02-4433-a426-0969a2fe1a51). Holes=1 overlaps=0 >[2018-08-18 10:05:05.324301] D [MSGID: 0] [dht-common.c:943:dht_lookup_dir_cbk] 0-vol0-dht: fixing assignment on /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m >[2018-08-18 10:05:05.324388] D [MSGID: 0] [dht-selfheal.c:1879:dht_selfheal_layout_new_directory] 0-vol0-dht: chunk size = 0xffffffff / 274638528 = 15.638619 >[2018-08-18 10:05:05.324414] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-15 >[2018-08-18 10:05:05.324429] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-16 >[2018-08-18 10:05:05.324444] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-17 >[2018-08-18 10:05:05.324458] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-18 >[2018-08-18 10:05:05.324472] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-19 >[2018-08-18 10:05:05.324486] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-2 >[2018-08-18 10:05:05.324499] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-20 >[2018-08-18 10:05:05.324520] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-21 >[2018-08-18 10:05:05.324534] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-22 >[2018-08-18 10:05:05.324547] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-23 >[2018-08-18 10:05:05.324561] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-24 >[2018-08-18 10:05:05.324575] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-25 >[2018-08-18 10:05:05.324588] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-26 >[2018-08-18 10:05:05.324602] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-27 >[2018-08-18 10:05:05.324615] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-28 >[2018-08-18 10:05:05.324629] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-29 >[2018-08-18 10:05:05.324642] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-3 >[2018-08-18 10:05:05.324656] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-30 >[2018-08-18 10:05:05.324669] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-31 >[2018-08-18 10:05:05.324682] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-32 >[2018-08-18 10:05:05.324696] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-33 >[2018-08-18 10:05:05.324709] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-34 >[2018-08-18 10:05:05.324724] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-35 >[2018-08-18 10:05:05.324738] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-4 >[2018-08-18 10:05:05.324751] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-5 >[2018-08-18 10:05:05.324764] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-6 >[2018-08-18 10:05:05.324777] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-7 >[2018-08-18 10:05:05.324790] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-8 >[2018-08-18 10:05:05.324810] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-9 >[2018-08-18 10:05:05.324827] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-0 >[2018-08-18 10:05:05.324840] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-1 >[2018-08-18 10:05:05.324854] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-10 >[2018-08-18 10:05:05.324872] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-11 >[2018-08-18 10:05:05.324886] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-12 >[2018-08-18 10:05:05.324900] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-13 >[2018-08-18 10:05:05.324913] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-14 >[2018-08-18 10:05:05.325290] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-vol0-dht: Found anomalies in /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m (gfid = b31240fd-5c02-4433-a426-0969a2fe1a51). Holes=1 overlaps=0 >[2018-08-18 10:05:05.325309] D [MSGID: 0] [dht-common.c:943:dht_lookup_dir_cbk] 0-vol0-dht: fixing assignment on /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m >[2018-08-18 10:05:05.325358] D [MSGID: 0] [dht-selfheal.c:1879:dht_selfheal_layout_new_directory] 0-vol0-dht: chunk size = 0xffffffff / 274638528 = 15.638619 >[2018-08-18 10:05:05.325370] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-15 >[2018-08-18 10:05:05.325377] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-16 >[2018-08-18 10:05:05.325384] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-17 >[2018-08-18 10:05:05.325390] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-18 >[2018-08-18 10:05:05.325398] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-19 >[2018-08-18 10:05:05.325405] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-2 >[2018-08-18 10:05:05.325411] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-20 >[2018-08-18 10:05:05.325417] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-21 >[2018-08-18 10:05:05.325423] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-22 >[2018-08-18 10:05:05.325429] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-23 >[2018-08-18 10:05:05.325435] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-24 >[2018-08-18 10:05:05.325441] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-25 >[2018-08-18 10:05:05.325446] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-26 >[2018-08-18 10:05:05.325453] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-27 >[2018-08-18 10:05:05.325459] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-28 >[2018-08-18 10:05:05.325465] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-29 >[2018-08-18 10:05:05.325475] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-3 >[2018-08-18 10:05:05.325485] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-30 >[2018-08-18 10:05:05.325491] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-31 >[2018-08-18 10:05:05.325493] D [MSGID: 0] [dht-selfheal.c:1879:dht_selfheal_layout_new_directory] 0-vol0-dht: chunk size = 0xffffffff / 274638528 = 15.638619 >[2018-08-18 10:05:05.325498] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-32 >[2018-08-18 10:05:05.325516] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-15 >[2018-08-18 10:05:05.325529] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-16 >[2018-08-18 10:05:05.325532] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-33 >[2018-08-18 10:05:05.325539] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-17 >[2018-08-18 10:05:05.325550] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-18 >[2018-08-18 10:05:05.325555] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-34 >[2018-08-18 10:05:05.325556] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-19 >[2018-08-18 10:05:05.325568] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-35 >[2018-08-18 10:05:05.325583] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-4 >[2018-08-18 10:05:05.325593] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-5 >[2018-08-18 10:05:05.325586] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-2 >[2018-08-18 10:05:05.325606] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-20 >[2018-08-18 10:05:05.325610] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-6 >[2018-08-18 10:05:05.325623] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-21 >[2018-08-18 10:05:05.325630] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-7 >[2018-08-18 10:05:05.325633] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-22 >[2018-08-18 10:05:05.325643] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-8 >[2018-08-18 10:05:05.325656] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-9 >[2018-08-18 10:05:05.325658] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-23 >[2018-08-18 10:05:05.325675] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-24 >[2018-08-18 10:05:05.325683] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-25 >[2018-08-18 10:05:05.325687] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-0 >[2018-08-18 10:05:05.325689] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-26 >[2018-08-18 10:05:05.325712] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-1 >[2018-08-18 10:05:05.325719] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-27 >[2018-08-18 10:05:05.325732] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-10 >[2018-08-18 10:05:05.325739] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-28 >[2018-08-18 10:05:05.325757] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-29 >[2018-08-18 10:05:05.325753] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-11 >[2018-08-18 10:05:05.325781] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-3 >[2018-08-18 10:05:05.325792] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-30 >[2018-08-18 10:05:05.325783] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-12 >[2018-08-18 10:05:05.325801] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-31 >[2018-08-18 10:05:05.325817] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-32 >[2018-08-18 10:05:05.325823] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-33 >[2018-08-18 10:05:05.325824] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-13 >[2018-08-18 10:05:05.325831] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-34 >[2018-08-18 10:05:05.325842] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-14 >[2018-08-18 10:05:05.325848] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-35 >[2018-08-18 10:05:05.325858] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-4 >[2018-08-18 10:05:05.325864] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-5 >[2018-08-18 10:05:05.325869] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-6 >[2018-08-18 10:05:05.325885] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-7 >[2018-08-18 10:05:05.325890] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-8 >[2018-08-18 10:05:05.325896] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-9 >[2018-08-18 10:05:05.325902] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-0 >[2018-08-18 10:05:05.325908] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-1 >[2018-08-18 10:05:05.325919] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-10 >[2018-08-18 10:05:05.325924] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-11 >[2018-08-18 10:05:05.325930] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-12 >[2018-08-18 10:05:05.325935] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-13 >[2018-08-18 10:05:05.325940] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-14 >[2018-08-18 10:05:05.326012] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 10:05:05.326079] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-0: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.326096] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c1c01c6f0, vol0-client-0 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.326113] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-2: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.326099] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-1: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.326122] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c1c01c6f0, vol0-client-2 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.326139] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c1c01c6f0, vol0-client-1 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.327439] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m on vol0-replicate-15 >[2018-08-18 10:05:05.327730] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.327935] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.327939] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.327971] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.327985] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m with op_ret -1 [No such file or directory] >[2018-08-18 10:05:05.327997] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m missing on subvol vol0-replicate-15 >[2018-08-18 10:05:05.328008] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:05.328109] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-66: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.328142] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c1c01c240, vol0-client-66 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.328150] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-68: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.328144] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-67: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.328185] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c1c01c240, vol0-client-68 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.328193] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c1c01c240, vol0-client-67 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.328287] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328319] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328364] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328396] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328399] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328422] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328433] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328458] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328464] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328459] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-0 >[2018-08-18 10:05:05.328468] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328486] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328492] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-1 >[2018-08-18 10:05:05.328515] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328540] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-2 >[2018-08-18 10:05:05.328560] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328589] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328597] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328620] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328635] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-3 >[2018-08-18 10:05:05.328620] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328675] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328696] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328728] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328745] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-4 >[2018-08-18 10:05:05.328769] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328767] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328789] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328808] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328833] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328854] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-5 >[2018-08-18 10:05:05.328860] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328874] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328911] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328928] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.328947] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-6 >[2018-08-18 10:05:05.328953] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329026] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329039] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329060] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329060] D [MSGID: 109036] [dht-common.c:9587:dht_log_new_layout_for_dir_selfheal] 0-vol0-dht: Setting layout of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m with [Subvol_name: vol0-replicate-0, Err: -1 , Start: 3459834763 , Stop: 3579139409 , Hash: 1 ], [Subvol_name: vol0-replicate-1, Err: -1 , Start: 3579139410 , Stop: 3698444056 , Hash: 1 ], [Subvol_name: vol0-replicate-10, Err: -1 , Start: 3698444057 , Stop: 3817748703 , Hash: 1 ], [Subvol_name: vol0-replicate-11, Err: -1 , Start: 3817748704 , Stop: 3937053350 , Hash: 1 ], [Subvol_name: vol0-replicate-12, Err: -1 , Start: 3937053351 , Stop: 4056357997 , Hash: 1 ], [Subvol_name: vol0-replicate-13, Err: -1 , Start: 4056357998 , Stop: 4175662644 , Hash: 1 ], [Subvol_name: vol0-replicate-14, Err: -1 , Start: 4175662645 , Stop: 4294967295 , Hash: 1 ], [Subvol_name: vol0-replicate-15, Err: -1 , Start: 0 , Stop: 119304646 , Hash: 1 ], [Subvol_name: vol0-replicate-16, Err: -1 , Start: 119304647 , Stop: 238609293 , Hash: 1 ], [Subvol_name: vol0-replicate-17, Err: -1 , Start: 238609294 , Stop: 357913940 , Hash: 1 ], [Subvol_name: vol0-replicate-18, Err: -1 , Start: 357913941 , Stop: 477218587 , Hash: 1 ], [Subvol_name: vol0-replicate-19, Err: -1 , Start: 477218588 , Stop: 596523234 , Hash: 1 ], [Subvol_name: vol0-replicate-2, Err: -1 , Start: 596523235 , Stop: 715827881 , Hash: 1 ], [Subvol_name: vol0-replicate-20, Err: -1 , Start: 715827882 , Stop: 835132528 , Hash: 1 ], [Subvol_name: vol0-replicate-21, Err: -1 , Start: 835132529 , Stop: 954437175 , Hash: 1 ], [Subvol_name: vol0-replicate-22, Err: -1 , Start: 954437176 , Stop: 1073741822 , Hash: 1 ], [Subvol_name: vol0-replicate-23, Err: -1 , Start: 1073741823 , Stop: 1193046469 , Hash: 1 ], [Subvol_name: vol0-replicate-24, Err: -1 , Start: 1193046470 , Stop: 1312351116 , Hash: 1 ], [Subvol_name: vol0-replicate-25, Err: -1 , Start: 1312351117 , Stop: 1431655763 , Hash: 1 ], [Subvol_name: vol0-replicate-26, Err: -1 , Start: 1431655764 , Stop: 1550960410 , Hash: 1 ], [Subvol_name: vol0-replicate-27, Err: -1 , Start: 1550960411 , Stop: 1670265057 , Hash: 1 ], [Subvol_name: vol0-replicate-28, Err: -1 , Start: 1670265058 , Stop: 1789569704 , Hash: 1 ], [Subvol_name: vol0-replicate-29, Err: -1 , Start: 1789569705 , Stop: 1908874351 , Hash: 1 ], [Subvol_name: vol0-replicate-3, Err: -1 , Start: 1908874352 , Stop: 2028178998 , Hash: 1 ], [Subvol_name: vol0-replicate-30, Err: -1 , Start: 2028178999 , Stop: 2147483645 , Hash: 1 ], [Subvol_name: vol0-replicate-31, Err: -1 , Start: 2147483646 , Stop: 2266788292 , Hash: 1 ], [Subvol_name: vol0-replicate-32, Err: -1 , Start: 2266788293 , Stop: 2386092939 , Hash: 1 ], [Subvol_name: vol0-replicate-33, Err: -1 , Start: 2386092940 , Stop: 2505397586 , Hash: 1 ], [Subvol_name: vol0-replicate-34, Err: -1 , Start: 2505397587 , Stop: 2624702233 , Hash: 1 ], [Subvol_name: vol0-replicate-35, Err: -1 , Start: 2624702234 , Stop: 2744006880 , Hash: 1 ], [Subvol_name: vol0-replicate-4, Err: -1 , Start: 2744006881 , Stop: 2863311527 , Hash: 1 ], [Subvol_name: vol0-replicate-5, Err: -1 , Start: 2863311528 , Stop: 2982616174 , Hash: 1 ], [Subvol_name: vol0-replicate-6, Err: -1 , Start: 2982616175 , Stop: 3101920821 , Hash: 1 ], [Subvol_name: vol0-replicate-7, Err: -1 , Start: 3101920822 , Stop: 3221225468 , Hash: 1 ], [Subvol_name: vol0-replicate-8, Err: -1 , Start: 3221225469 , Stop: 3340530115 , Hash: 1 ], [Subvol_name: vol0-replicate-9, Err: -1 , Start: 3340530116 , Stop: 3459834762 , Hash: 1 ], >[2018-08-18 10:05:05.329084] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329101] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-7 >[2018-08-18 10:05:05.329105] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329113] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329163] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329170] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329168] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329204] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329217] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-9 >[2018-08-18 10:05:05.329219] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329240] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-8 >[2018-08-18 10:05:05.329243] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329241] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329297] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329299] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-0: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.329303] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329321] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329328] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329342] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329348] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329357] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-10 >[2018-08-18 10:05:05.329343] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-11 >[2018-08-18 10:05:05.329375] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329387] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-1: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.329388] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329429] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.329426] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329449] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329453] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-12 >[2018-08-18 10:05:05.329453] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329476] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329496] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329505] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-13 >[2018-08-18 10:05:05.329552] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329567] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.329577] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 10:05:05.329581] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329584] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329614] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329608] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 10:05:05.329626] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-14 >[2018-08-18 10:05:05.329633] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 10:05:05.329635] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329660] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329696] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-12: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.329764] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329777] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.329777] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329808] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329812] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 10:05:05.329829] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329847] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-16 >[2018-08-18 10:05:05.329856] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329861] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.329867] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329877] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329905] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329916] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-17 >[2018-08-18 10:05:05.329920] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329937] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329944] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 10:05:05.329943] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329975] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329992] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.329987] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 10:05:05.329997] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-18 >[2018-08-18 10:05:05.330017] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330023] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330061] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-15: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.330063] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330063] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330086] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-19 >[2018-08-18 10:05:05.330090] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.330111] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330131] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-17: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.330148] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330182] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330186] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:05.330197] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330199] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-20 >[2018-08-18 10:05:05.330202] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330259] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330259] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-18: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.330275] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330297] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-21 >[2018-08-18 10:05:05.330336] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-19: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.330334] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330345] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 10:05:05.330354] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 10:05:05.330350] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:05.330360] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330361] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330397] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330408] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-22 >[2018-08-18 10:05:05.330422] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-2: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.330452] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330456] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330495] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 10:05:05.330501] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330506] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330533] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330541] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-20: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.330534] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330568] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330570] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 10:05:05.330573] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330572] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330590] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330620] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.330610] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-23 >[2018-08-18 10:05:05.330631] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 10:05:05.330610] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-24 >[2018-08-18 10:05:05.330643] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330614] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330682] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330689] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330726] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330709] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330749] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-22: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.330769] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-25 >[2018-08-18 10:05:05.330773] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330784] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-20: lk op is for a transaction >[2018-08-18 10:05:05.330771] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330808] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-26 >[2018-08-18 10:05:05.330823] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330828] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330842] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:05.330842] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330847] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-23: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.330872] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330872] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330889] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-27 >[2018-08-18 10:05:05.330915] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330931] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-28 >[2018-08-18 10:05:05.330965] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330967] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.330967] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.330991] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-22: lk op is for a transaction >[2018-08-18 10:05:05.331001] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331007] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-25: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.331020] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331036] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-29 >[2018-08-18 10:05:05.331056] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331063] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331064] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331091] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-23: lk op is for a transaction >[2018-08-18 10:05:05.331106] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331112] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-26: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.331108] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331140] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331142] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-30 >[2018-08-18 10:05:05.331162] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-27: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.331151] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331186] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331188] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331211] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-31 >[2018-08-18 10:05:05.331215] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331240] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331251] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 10:05:05.331262] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331244] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-28: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.331274] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-32 >[2018-08-18 10:05:05.331260] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 10:05:05.331296] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331314] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-29: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.331313] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331317] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331297] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331363] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331361] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331369] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-26: lk op is for a transaction >[2018-08-18 10:05:05.331360] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331400] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-3: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.331379] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-33 >[2018-08-18 10:05:05.331405] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-27: lk op is for a transaction >[2018-08-18 10:05:05.331409] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-15 >[2018-08-18 10:05:05.331402] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331411] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331438] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331452] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331468] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331471] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331491] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 10:05:05.331494] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331494] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-34 >[2018-08-18 10:05:05.331509] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-35 >[2018-08-18 10:05:05.331527] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-15 cached_subvol null >[2018-08-18 10:05:05.331534] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-30: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.331539] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m >[2018-08-18 10:05:05.331557] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331557] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:05.331568] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331582] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331593] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331606] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.331659] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 10:05:05.331677] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.331719] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-32: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.331765] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-30: lk op is for a transaction >[2018-08-18 10:05:05.331797] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-33: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.331876] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-34: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.331921] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:05.331945] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 10:05:05.332028] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 10:05:05.332076] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.332094] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 10:05:05.332144] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-5: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.332202] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-6: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.332284] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.332292] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 10:05:05.332388] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 10:05:05.332393] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-8: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.332389] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-35: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.332439] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 10:05:05.332446] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0 returned with op_ret 0 >[2018-08-18 10:05:05.332491] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-9: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.332518] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 10:05:05.332669] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 10:05:05.332715] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 10:05:05.332842] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 10:05:05.334916] D [MSGID: 0] [dht-helper.c:1105:dht_subvol_get_hashed] 0-vol0-dht: Missing layout. path=/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a, parent gfid =b31240fd-5c02-4433-a426-0969a2fe1a51 >[2018-08-18 10:05:05.334950] D [MSGID: 0] [dht-common.c:2716:dht_lookup] 0-vol0-dht: no subvolume in layout for path=/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a, checking on all the subvols to see if it is a directory >[2018-08-18 10:05:05.334962] D [MSGID: 0] [dht-common.c:2730:dht_lookup] 0-vol0-dht: Found null hashed subvol. Calling lookup on all nodes. >[2018-08-18 10:05:05.335262] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335298] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335313] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335335] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335334] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335354] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335364] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-0 returned error [No such file or directory] >[2018-08-18 10:05:05.335365] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335391] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335397] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335403] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-1 returned error [No such file or directory] >[2018-08-18 10:05:05.335418] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335456] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335470] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335482] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-2 returned error [No such file or directory] >[2018-08-18 10:05:05.335485] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335485] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335528] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335542] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335559] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-3 returned error [No such file or directory] >[2018-08-18 10:05:05.335590] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335597] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335640] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335657] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335669] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-4 returned error [No such file or directory] >[2018-08-18 10:05:05.335680] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335710] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335716] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335745] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335768] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-5 returned error [No such file or directory] >[2018-08-18 10:05:05.335786] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335820] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335837] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335859] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335873] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-6 returned error [No such file or directory] >[2018-08-18 10:05:05.335916] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335942] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335959] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335978] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.335988] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-7 returned error [No such file or directory] >[2018-08-18 10:05:05.335992] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 10:05:05.336000] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0 returned with op_ret 0" repeated 35 times between [2018-08-18 10:05:05.332446] and [2018-08-18 10:05:05.335696] >[2018-08-18 10:05:05.335991] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m on vol0-replicate-15 >[2018-08-18 10:05:05.336023] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336048] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336066] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336080] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336106] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336109] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336130] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336140] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336134] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-8 returned error [No such file or directory] >[2018-08-18 10:05:05.336157] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336193] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-9 returned error [No such file or directory] >[2018-08-18 10:05:05.336198] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336219] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336229] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-10 returned error [No such file or directory] >[2018-08-18 10:05:05.336291] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336306] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336336] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336355] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336363] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336357] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336357] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336396] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336415] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336413] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336433] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336434] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-11 returned error [No such file or directory] >[2018-08-18 10:05:05.336415] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336447] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m with op_ret -1 [No such file or directory] >[2018-08-18 10:05:05.336453] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-12 returned error [No such file or directory] >[2018-08-18 10:05:05.336465] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m missing on subvol vol0-replicate-15 >[2018-08-18 10:05:05.336476] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:05.336472] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336473] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336522] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336526] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336544] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336560] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336556] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336576] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-13 returned error [No such file or directory] >[2018-08-18 10:05:05.336597] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336599] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336598] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336620] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336623] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-14 returned error [No such file or directory] >[2018-08-18 10:05:05.336649] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336665] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-15 returned error [No such file or directory] >[2018-08-18 10:05:05.336705] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336713] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336749] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336752] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336771] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336773] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336804] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336805] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-16 returned error [No such file or directory] >[2018-08-18 10:05:05.336805] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336821] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336839] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336852] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-17 returned error [No such file or directory] >[2018-08-18 10:05:05.336876] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336873] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336891] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336887] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336904] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-0 >[2018-08-18 10:05:05.336918] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336920] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336921] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336944] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336962] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-18 returned error [No such file or directory] >[2018-08-18 10:05:05.336973] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336984] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336987] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336991] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336988] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337002] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.336989] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337047] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337051] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337058] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337048] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-1 >[2018-08-18 10:05:05.337052] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337072] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-19 returned error [No such file or directory] >[2018-08-18 10:05:05.337084] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337061] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337094] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-2 >[2018-08-18 10:05:05.337096] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337123] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337132] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337131] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337130] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337144] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337149] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337128] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337164] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337176] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337187] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337189] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-3 >[2018-08-18 10:05:05.337163] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337195] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-20 returned error [No such file or directory] >[2018-08-18 10:05:05.337203] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337210] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337236] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-21 returned error [No such file or directory] >[2018-08-18 10:05:05.337240] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337208] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337256] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337257] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337258] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-22 returned error [No such file or directory] >[2018-08-18 10:05:05.337270] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-4 >[2018-08-18 10:05:05.337325] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337330] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337361] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337374] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337387] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337397] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-23 returned error [No such file or directory] >[2018-08-18 10:05:05.337401] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337422] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337430] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337441] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337449] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-5 >[2018-08-18 10:05:05.337457] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337461] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337467] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337470] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337484] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337494] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-6 >[2018-08-18 10:05:05.337496] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337506] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-24 returned error [No such file or directory] >[2018-08-18 10:05:05.337511] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337515] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337530] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337563] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337551] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337576] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337594] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337598] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337597] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337614] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-25 returned error [No such file or directory] >[2018-08-18 10:05:05.337601] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337616] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337630] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337633] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-7 >[2018-08-18 10:05:05.337639] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337652] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337645] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337662] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-8 >[2018-08-18 10:05:05.337669] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337679] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-26 returned error [No such file or directory] >[2018-08-18 10:05:05.337675] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337692] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337707] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337720] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337727] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-9 >[2018-08-18 10:05:05.337711] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337711] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337711] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337761] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337769] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-27 returned error [No such file or directory] >[2018-08-18 10:05:05.337779] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337791] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337820] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337828] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337841] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337849] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337864] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-10 >[2018-08-18 10:05:05.337888] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337902] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337889] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337912] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337928] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337932] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337946] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-28 returned error [No such file or directory] >[2018-08-18 10:05:05.337967] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337967] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337988] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337992] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.337990] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-29 returned error [No such file or directory] >[2018-08-18 10:05:05.338003] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338022] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338001] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338048] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-11 >[2018-08-18 10:05:05.338058] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338053] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338062] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338078] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338084] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-30 returned error [No such file or directory] >[2018-08-18 10:05:05.338092] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338094] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338100] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338124] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338135] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338140] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-31 returned error [No such file or directory] >[2018-08-18 10:05:05.338146] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338150] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338133] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338158] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338160] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338174] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338172] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-12 >[2018-08-18 10:05:05.338192] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338196] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338202] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-32 returned error [No such file or directory] >[2018-08-18 10:05:05.338207] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-13 >[2018-08-18 10:05:05.338221] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338228] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338236] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338252] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338255] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338261] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-14 >[2018-08-18 10:05:05.338343] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338370] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338371] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338373] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338390] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338405] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338409] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338410] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338436] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-34 returned error [No such file or directory] >[2018-08-18 10:05:05.338443] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338443] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338449] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338443] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338464] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-33 returned error [No such file or directory] >[2018-08-18 10:05:05.338469] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338466] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338486] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338477] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338496] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-16 >[2018-08-18 10:05:05.338509] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338522] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-35 returned error [No such file or directory] >[2018-08-18 10:05:05.338526] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338535] D [MSGID: 0] [dht-common.c:964:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338546] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338567] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338577] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338583] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338588] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338598] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338620] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338633] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-17 >[2018-08-18 10:05:05.338637] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338638] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338662] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338674] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338683] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-18 >[2018-08-18 10:05:05.338695] D [MSGID: 0] [dht-helper.c:1105:dht_subvol_get_hashed] 0-vol0-dht: Missing layout. path=/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a, parent gfid =b31240fd-5c02-4433-a426-0969a2fe1a51 >[2018-08-18 10:05:05.338717] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338722] E [MSGID: 109011] [dht-common.c:7248:dht_create] 0-vol0-dht: no subvolume in layout for path=/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a >[2018-08-18 10:05:05.338719] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338740] D [MSGID: 0] [dht-common.c:7331:dht_create] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-dht returned -1 error: Input/output error [Input/output error] >[2018-08-18 10:05:05.338748] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338751] D [MSGID: 0] [md-cache.c:1703:mdc_create_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-md-cache returned -1 error: Input/output error [Input/output error] >[2018-08-18 10:05:05.338767] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338767] D [MSGID: 0] [io-stats.c:1941:io_stats_create_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0 returned -1 error: Input/output error [Input/output error] >[2018-08-18 10:05:05.338775] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-19 >[2018-08-18 10:05:05.338816] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338835] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338860] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338878] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338895] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-20 >[2018-08-18 10:05:05.338913] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338926] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.338997] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339019] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339027] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339040] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339039] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-21 >[2018-08-18 10:05:05.339056] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339074] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339083] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-22 >[2018-08-18 10:05:05.339170] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339165] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339166] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339216] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339238] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-23 >[2018-08-18 10:05:05.339226] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339252] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339311] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339337] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339341] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339367] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-24 >[2018-08-18 10:05:05.339390] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339399] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339414] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339423] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339442] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-25 >[2018-08-18 10:05:05.339450] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339456] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339519] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339486] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339523] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339537] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339539] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-26 >[2018-08-18 10:05:05.339559] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339570] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-27 >[2018-08-18 10:05:05.339643] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339652] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339688] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339708] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339719] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-28 >[2018-08-18 10:05:05.339740] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339774] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339795] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339817] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339828] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-29 >[2018-08-18 10:05:05.339874] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339861] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339860] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339907] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339915] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339917] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339908] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339940] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339943] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-30 >[2018-08-18 10:05:05.339953] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-31 >[2018-08-18 10:05:05.339980] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.339987] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.340053] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.340078] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.340089] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-32 >[2018-08-18 10:05:05.340104] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.340121] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.340119] D [MSGID: 0] [dht-selfheal.c:1206:dht_selfheal_dir_mkdir_setacl] 0-vol0-dht: ACL_DEFAULT xattr not present >[2018-08-18 10:05:05.340143] D [MSGID: 0] [dht-selfheal.c:1219:dht_selfheal_dir_mkdir_setacl] 0-vol0-dht: ACL_ACCESS xattr not present >[2018-08-18 10:05:05.340142] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.340167] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-1 >[2018-08-18 10:05:05.340174] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.340187] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-33 >[2018-08-18 10:05:05.340234] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.340248] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.340257] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.340264] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-10 >[2018-08-18 10:05:05.340261] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.340277] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.340290] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-34 >[2018-08-18 10:05:05.340292] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.340314] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-15 >[2018-08-18 10:05:05.340322] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.340332] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-11 >[2018-08-18 10:05:05.340350] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.340365] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.340386] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.340387] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-1: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.340395] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m) from subvol vol0-replicate-35 >[2018-08-18 10:05:05.340405] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-15 cached_subvol null >[2018-08-18 10:05:05.340411] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m >[2018-08-18 10:05:05.340417] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.340423] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.340422] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-12 >[2018-08-18 10:05:05.340431] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.340439] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.340446] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.340441] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-10: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.340499] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-16 >[2018-08-18 10:05:05.340518] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-3: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.340522] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-4: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.340542] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c54016f40, vol0-client-4 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.340544] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c54016f40, vol0-client-3 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.340543] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-5: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.340545] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-11: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.340560] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-30: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.340592] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5402f250, vol0-client-30 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.340596] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-31: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.340598] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c54016f40, vol0-client-5 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.340593] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-32: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.340610] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5402f250, vol0-client-31 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.340604] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-17 >[2018-08-18 10:05:05.340621] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 10:05:05.340622] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5402f250, vol0-client-32 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.340641] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-12: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.340644] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 10:05:05.340683] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-16: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.340691] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-34: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.340704] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-18 >[2018-08-18 10:05:05.340712] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c540055d0, vol0-client-34 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.340731] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-35: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.340732] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-33: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.340748] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c540055d0, vol0-client-35 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.340755] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c540055d0, vol0-client-33 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.340769] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 10:05:05.340768] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-0': avail_percent is: 96.00 and avail_space is: 7683154767872 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.340778] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-36: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.340789] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5401b2f0, vol0-client-36 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.340793] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-37: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.340809] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-1': avail_percent is: 95.00 and avail_space is: 7649950310400 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.340812] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5401b2f0, vol0-client-37 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.340814] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-38: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.340831] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-1 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.340832] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-19 >[2018-08-18 10:05:05.340837] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-17: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.340837] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5401b2f0, vol0-client-38 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.340860] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-2': avail_percent is: 96.00 and avail_space is: 7682652368896 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.340840] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-49: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.340879] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.340882] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-48: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.340880] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 10:05:05.340899] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-3': avail_percent is: 96.00 and avail_space is: 7679958376448 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.340909] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c54008490, vol0-client-48 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.340914] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c54008490, vol0-client-49 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.340923] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-10 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.340931] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-4': avail_percent is: 96.00 and avail_space is: 7680306479104 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.340925] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-50: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.340951] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.340960] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c54008490, vol0-client-50 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.340975] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:05.340977] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-6': avail_percent is: 94.00 and avail_space is: 7572583333888 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.340972] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-11 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.340988] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-52: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.340973] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-18: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.341006] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-51: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341004] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5401d820, vol0-client-52 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341007] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-5': avail_percent is: 95.00 and avail_space is: 7674803892224 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341001] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.341037] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-53: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341051] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-7': avail_percent is: 95.00 and avail_space is: 7602560430080 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341056] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5401d820, vol0-client-53 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341023] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5401d820, vol0-client-51 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341071] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-8': avail_percent is: 95.00 and avail_space is: 7674537177088 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341065] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-24 >[2018-08-18 10:05:05.341077] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-12 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341102] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 10:05:05.341111] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.341147] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-9': avail_percent is: 95.00 and avail_space is: 7675655356416 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341144] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-16 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341152] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-54: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341146] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-19: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.341176] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-10': avail_percent is: 94.00 and avail_space is: 7587969724416 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341173] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-3 >[2018-08-18 10:05:05.341204] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-11': avail_percent is: 95.00 and avail_space is: 7666756444160 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341175] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5401bd20, vol0-client-54 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341199] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-55: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341201] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.341230] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5401bd20, vol0-client-55 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341240] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-56: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341260] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5401bd20, vol0-client-56 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341264] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-12': avail_percent is: 95.00 and avail_space is: 7662852456448 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341266] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-17 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341273] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-31 >[2018-08-18 10:05:05.341286] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.341293] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-13': avail_percent is: 95.00 and avail_space is: 7672171716608 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341274] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 10:05:05.341309] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-57: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341314] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-59: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341316] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-24: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.341332] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c54001240, vol0-client-59 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341342] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-14': avail_percent is: 95.00 and avail_space is: 7671006871552 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341341] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c54001240, vol0-client-57 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341369] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-58: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341379] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c54001240, vol0-client-58 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341393] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 10:05:05.341396] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-33 >[2018-08-18 10:05:05.341379] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-3: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.341423] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-15': avail_percent is: 95.00 and avail_space is: 7657355005952 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341443] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-16': avail_percent is: 96.00 and avail_space is: 7681130377216 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341456] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-72: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341463] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-18 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341463] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-17': avail_percent is: 95.00 and avail_space is: 7670377443328 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341472] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c540055d0, vol0-client-72 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341474] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-73: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341494] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-74: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341522] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-18': avail_percent is: 95.00 and avail_space is: 7620654678016 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341516] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c540055d0, vol0-client-74 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341534] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.341521] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c540055d0, vol0-client-73 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341554] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-19': avail_percent is: 95.00 and avail_space is: 7661022814208 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341527] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-31: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.341519] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-34 >[2018-08-18 10:05:05.341572] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 10:05:05.341549] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-9: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341591] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5401b2f0, vol0-client-9 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341586] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-11: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341587] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-10: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341609] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-21': avail_percent is: 95.00 and avail_space is: 7663950176256 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341611] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5401b2f0, vol0-client-11 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341617] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5401b2f0, vol0-client-10 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341628] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-19 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341638] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 10:05:05.341641] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.341644] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-35 >[2018-08-18 10:05:05.341659] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-22': avail_percent is: 95.00 and avail_space is: 7668892258304 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341642] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-33: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.341696] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-20': avail_percent is: 95.00 and avail_space is: 7611642155008 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341700] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-94: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341701] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-93: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341709] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-95: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341723] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-23': avail_percent is: 95.00 and avail_space is: 7670234238976 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341729] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c54008490, vol0-client-93 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341731] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c54008490, vol0-client-94 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341748] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-24': avail_percent is: 95.00 and avail_space is: 7665744367616 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341746] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c54008490, vol0-client-95 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341759] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-5 >[2018-08-18 10:05:05.341775] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-25': avail_percent is: 95.00 and avail_space is: 7650961227776 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341773] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:05.341798] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-34: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.341799] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-24 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341823] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.341831] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-26': avail_percent is: 95.00 and avail_space is: 7659003944960 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341848] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-100: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341850] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-99: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341864] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-3 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341849] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-8 >[2018-08-18 10:05:05.341881] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5401d820, vol0-client-99 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341885] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-101: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341883] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.341907] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5401d820, vol0-client-101 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341907] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-27': avail_percent is: 95.00 and avail_space is: 7652756992000 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341886] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5401d820, vol0-client-100 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341930] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-28': avail_percent is: 95.00 and avail_space is: 7652597972992 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341932] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-102: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341935] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 10:05:05.341955] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5401bd20, vol0-client-102 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341934] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-5: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.341961] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-104: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341979] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-103: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.341980] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-31 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.341988] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-30': avail_percent is: 95.00 and avail_space is: 7677991510016 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.341987] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5401bd20, vol0-client-104 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.342006] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.342008] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5401bd20, vol0-client-103 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.342015] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-31': avail_percent is: 95.00 and avail_space is: 7678388396032 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.342040] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 10:05:05.342075] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-29': avail_percent is: 95.00 and avail_space is: 7639883448320 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.342078] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-15: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.342075] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-35: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.342092] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5402f250, vol0-client-15 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.342100] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-32': avail_percent is: 95.00 and avail_space is: 7664158932992 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.342107] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-16: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.342135] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5402f250, vol0-client-16 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.342137] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-8: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.342139] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-33': avail_percent is: 95.00 and avail_space is: 7679374594048 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.342144] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-17: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.342161] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-34': avail_percent is: 95.00 and avail_space is: 7668156858368 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.342155] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-33 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.342173] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5402f250, vol0-client-17 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.342184] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-35': avail_percent is: 95.00 and avail_space is: 7666712231936 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.342194] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 10:05:05.342196] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.342208] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-34 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.342216] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-105: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.342237] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c54001240, vol0-client-105 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.342242] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-24: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.342248] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.342258] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c540055d0, vol0-client-24 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.342244] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-106: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.342284] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-26: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.342286] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c54001240, vol0-client-106 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.342287] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-107: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.342287] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-25: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.342314] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c54001240, vol0-client-107 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.342322] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c540055d0, vol0-client-25 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.342305] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c540055d0, vol0-client-26 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.342342] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 10:05:05.342344] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 10:05:05.342381] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-5 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.342400] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.342501] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-15: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.342524] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-35 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.342535] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-8 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.342547] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.344562] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 10:05:05.342557] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.344561] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-0: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.344797] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 10:05:05.344797] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.344845] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.344856] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-1: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.344904] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.344949] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.345008] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-15: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.345026] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-12: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.345088] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 10:05:05.345111] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 10:05:05.345132] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 10:05:05.345139] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 10:05:05.345207] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:05.345261] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 10:05:05.345294] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 10:05:05.345445] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.345498] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-17: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.345653] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-18: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.345662] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-2: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.345677] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:05.345723] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-19: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.345725] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 10:05:05.345733] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-20: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.345811] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.345857] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-22: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.345857] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 10:05:05.345866] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-23: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.345946] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 10:05:05.346018] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-20: lk op is for a transaction >[2018-08-18 10:05:05.346012] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-25: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.346018] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 10:05:05.346119] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-26: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.346132] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:05.346153] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-22: lk op is for a transaction >[2018-08-18 10:05:05.346177] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-23: lk op is for a transaction >[2018-08-18 10:05:05.346225] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-28: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.346214] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-27: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.346262] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 10:05:05.346279] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.346309] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-26: lk op is for a transaction >[2018-08-18 10:05:05.346329] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-29: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.346422] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 10:05:05.346470] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-30: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.346516] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-27: lk op is for a transaction >[2018-08-18 10:05:05.346525] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 10:05:05.346580] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:05.346625] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-3: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.346671] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-32: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.346674] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-30: lk op is for a transaction >[2018-08-18 10:05:05.346843] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 10:05:05.346871] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.346922] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 10:05:05.347005] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.347014] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-33: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.347092] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-34: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.347117] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:05.347180] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-6: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.347246] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.347284] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 10:05:05.347291] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 10:05:05.347344] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 10:05:05.347403] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-5: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.347433] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 10:05:05.347477] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-9: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.347511] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 10:05:05.347645] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 10:05:05.347655] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-8: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.347689] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 10:05:05.347692] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-35: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.347858] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 10:05:05.348018] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 10:05:05.348118] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-0: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.348145] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c4c09fb90, vol0-client-0 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.348126] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-2: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.348150] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-1: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.348162] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c4c09fb90, vol0-client-2 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.348177] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c4c09fb90, vol0-client-1 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.352852] D [MSGID: 0] [dht-selfheal.c:1206:dht_selfheal_dir_mkdir_setacl] 0-vol0-dht: ACL_DEFAULT xattr not present >[2018-08-18 10:05:05.352874] D [MSGID: 0] [dht-selfheal.c:1219:dht_selfheal_dir_mkdir_setacl] 0-vol0-dht: ACL_ACCESS xattr not present >[2018-08-18 10:05:05.352885] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-1 >[2018-08-18 10:05:05.352953] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-10 >[2018-08-18 10:05:05.353006] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-11 >[2018-08-18 10:05:05.353064] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-12 >[2018-08-18 10:05:05.353119] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-16 >[2018-08-18 10:05:05.353135] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-1: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.353145] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-10: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.353179] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-24 >[2018-08-18 10:05:05.353200] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-11: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.353268] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-3: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353273] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-3 >[2018-08-18 10:05:05.353274] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-4: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353311] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58053350, vol0-client-4 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353287] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58053350, vol0-client-3 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353286] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-12: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.353316] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-5: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353321] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-16: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.353345] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-30: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353353] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58053350, vol0-client-5 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353360] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-31: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353364] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58055f30, vol0-client-30 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353381] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58055f30, vol0-client-31 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353384] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 10:05:05.353381] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-31 >[2018-08-18 10:05:05.353393] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-32: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353415] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58055f30, vol0-client-32 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353416] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-33: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353427] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-34: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353432] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5845cdc0, vol0-client-33 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353433] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-35: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353441] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 10:05:05.353465] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-36: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353477] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-24: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.353489] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58055310, vol0-client-36 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353465] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5845cdc0, vol0-client-35 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353474] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-34 >[2018-08-18 10:05:05.353447] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c5845cdc0, vol0-client-34 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353514] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-48: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353524] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58048f10, vol0-client-48 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353525] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 10:05:05.353526] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-37: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353558] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58055310, vol0-client-37 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353565] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-3: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.353574] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-38: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353574] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-49: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353587] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-50: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353581] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-35 >[2018-08-18 10:05:05.353600] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58048f10, vol0-client-49 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353604] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58048f10, vol0-client-50 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353588] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58055310, vol0-client-38 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353623] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:05.353633] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-31: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.353633] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 10:05:05.353633] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-72: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353641] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-73: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353672] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58054b00, vol0-client-73 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353672] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58054b00, vol0-client-72 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353676] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-5 >[2018-08-18 10:05:05.353700] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-10 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353704] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-74: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353719] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58054b00, vol0-client-74 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353721] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.353724] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-9: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353731] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 10:05:05.353745] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-10: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353750] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-1 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353760] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58005560, vol0-client-10 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353743] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58005560, vol0-client-9 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353784] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on subvol vol0-replicate-8 >[2018-08-18 10:05:05.353766] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-34: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.353767] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.353812] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-11: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353813] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-11 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353830] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58005560, vol0-client-11 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353843] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 10:05:05.353841] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.353842] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-93: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353861] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-94: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353862] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58018770, vol0-client-93 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353878] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58018770, vol0-client-94 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353888] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-95: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353891] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-16 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353905] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58018770, vol0-client-95 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353911] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-35: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.353915] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.353926] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-5: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.353916] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-12 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353924] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:05.353952] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.353961] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-24 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.353971] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-103: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353970] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-102: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.353985] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.354007] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c580074c0, vol0-client-103 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.354005] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c580074c0, vol0-client-102 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.354004] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-104: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.354026] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-3 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.354036] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c580074c0, vol0-client-104 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.354035] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-8: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.354039] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-106: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.354041] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-107: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.354064] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 10:05:05.354053] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-15: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.354066] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58052b40, vol0-client-106 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.354065] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.354079] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58052b40, vol0-client-107 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.354088] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c580542f0, vol0-client-15 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.354111] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-105: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.354122] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58052b40, vol0-client-105 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.354122] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-16: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.354123] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-17: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.354138] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-31 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.354151] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c580542f0, vol0-client-16 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.354153] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c580542f0, vol0-client-17 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.354152] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.354154] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 10:05:05.354177] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 10:05:05.354182] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-25: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.354196] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-24: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.354206] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58055f30, vol0-client-25 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.354222] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58055f30, vol0-client-24 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.354207] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-26: remote operation failed. Path: /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m [File exists] >[2018-08-18 10:05:05.354238] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-34 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.354256] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.354256] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c58055f30, vol0-client-26 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.354281] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 10:05:05.354327] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-35 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.354364] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-5 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.354365] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.354448] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 10:05:05.354395] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.354447] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c3405cbd0, vol0-replicate-8 returned -1 error: File exists [File exists] >[2018-08-18 10:05:05.354492] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m, gfid = b31240fd-5c02-4433-a426-0969a2fe1a51 [File exists] >[2018-08-18 10:05:05.357077] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-0: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.357307] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 10:05:05.357477] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.357497] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-1: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.357547] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.357650] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.357669] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 10:05:05.357673] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.357702] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 10:05:05.357723] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-12: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.357774] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-15: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.357792] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 10:05:05.357882] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 10:05:05.357888] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:05.357931] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 10:05:05.357970] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 10:05:05.358026] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-17: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.358150] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-18: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.358197] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.358227] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-19: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.358282] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 10:05:05.358333] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-2: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.358360] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 10:05:05.358387] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:05.358421] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 10:05:05.358445] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-20: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.358505] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.358542] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 10:05:05.358596] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-22: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.358669] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-20: lk op is for a transaction >[2018-08-18 10:05:05.358675] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-23: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.358691] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:05.358779] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-22: lk op is for a transaction >[2018-08-18 10:05:05.358863] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-25: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.358880] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-23: lk op is for a transaction >[2018-08-18 10:05:05.358952] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-26: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.359042] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-27: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.359049] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.359061] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 10:05:05.359139] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-28: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.359152] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-26: lk op is for a transaction >[2018-08-18 10:05:05.359240] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-29: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.359261] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 10:05:05.359276] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-27: lk op is for a transaction >[2018-08-18 10:05:05.359326] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 10:05:05.359439] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:05.359468] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-30: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.359637] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-3: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.359638] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-32: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.359687] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-33: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.359696] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-30: lk op is for a transaction >[2018-08-18 10:05:05.359750] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.359848] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 10:05:05.359886] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 10:05:05.359906] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 10:05:05.359961] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:05.360012] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.360079] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-34: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.360199] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-35: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.360219] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-6: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.360236] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 10:05:05.360293] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 10:05:05.360314] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.360346] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-5: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.360383] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 10:05:05.360450] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 10:05:05.360516] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 10:05:05.360521] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-9: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.360571] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 10:05:05.360710] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 10:05:05.360733] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-8: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.361022] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 10:05:05.361122] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-1: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.361127] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-0: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.361139] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c58079d60, vol0-client-1 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.361155] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c58079d60, vol0-client-0 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.361142] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-2: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.361180] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c58079d60, vol0-client-2 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.361410] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-0: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.361490] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-1: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.361534] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-2: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.361599] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-3: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.361720] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-4: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.361785] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-5: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.361857] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 10:05:05.361877] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 10:05:05.361922] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-6: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.362049] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-7: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.362057] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-8: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.362092] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-9: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.362219] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-10: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.362382] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-12: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.362404] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-11: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.362446] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-13: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.362471] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 10:05:05.362482] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-14: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.362565] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-16: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.362638] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-17: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.362735] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-18: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.362838] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-19: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.362873] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-20: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.362969] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-21: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.363020] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-22: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.363098] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-23: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.363155] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-24: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.363228] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-25: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.363299] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-26: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.363364] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-27: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.363450] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-28: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.363534] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-29: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.363620] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-26: lk op is for a transaction >[2018-08-18 10:05:05.363620] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-30: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.363691] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-31: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.363685] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-27: lk op is for a transaction >[2018-08-18 10:05:05.363767] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 10:05:05.363790] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-32: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.363892] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-33: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.363961] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 10:05:05.364005] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-34: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.364078] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-35: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.364188] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 10:05:05.367663] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a on vol0-replicate-29 >[2018-08-18 10:05:05.367918] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.367928] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.367931] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.367981] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.367997] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a with op_ret -1 [No such file or directory] >[2018-08-18 10:05:05.368011] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a missing on subvol vol0-replicate-29 >[2018-08-18 10:05:05.368022] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:05.368287] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368310] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368316] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368346] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368351] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368361] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-0 >[2018-08-18 10:05:05.368405] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368421] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368434] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368444] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368449] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368462] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-1 >[2018-08-18 10:05:05.368465] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368490] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368502] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-2 >[2018-08-18 10:05:05.368544] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368548] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368565] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368590] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368603] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-3 >[2018-08-18 10:05:05.368652] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368674] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368677] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368703] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368717] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-4 >[2018-08-18 10:05:05.368750] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368800] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368808] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368848] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368851] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368868] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-5 >[2018-08-18 10:05:05.368875] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368928] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368945] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368947] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368965] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368956] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.368981] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-6 >[2018-08-18 10:05:05.369000] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369011] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369019] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-7 >[2018-08-18 10:05:05.369019] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369067] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369098] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369115] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-8 >[2018-08-18 10:05:05.369120] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369158] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369155] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369197] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369211] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-9 >[2018-08-18 10:05:05.369224] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369262] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369263] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369297] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369309] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-10 >[2018-08-18 10:05:05.369338] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369367] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369378] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369391] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369405] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369419] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-11 >[2018-08-18 10:05:05.369454] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369491] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369500] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369522] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369551] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369553] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369573] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369573] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-12 >[2018-08-18 10:05:05.369590] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-13 >[2018-08-18 10:05:05.369607] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369612] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369608] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369645] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369655] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-14 >[2018-08-18 10:05:05.369669] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369701] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369732] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369745] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369755] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-15 >[2018-08-18 10:05:05.369803] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369813] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369870] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369905] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369909] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369916] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369933] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-16 >[2018-08-18 10:05:05.369945] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369967] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.369978] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-17 >[2018-08-18 10:05:05.369996] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370029] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370029] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370076] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370092] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370106] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370114] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-18 >[2018-08-18 10:05:05.370116] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370143] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370158] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-19 >[2018-08-18 10:05:05.370219] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370258] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370252] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370307] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370318] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370313] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370333] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-20 >[2018-08-18 10:05:05.370367] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370386] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370395] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-21 >[2018-08-18 10:05:05.370421] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370437] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370433] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370470] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370486] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-22 >[2018-08-18 10:05:05.370486] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370500] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370540] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370566] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370574] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370583] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-23 >[2018-08-18 10:05:05.370601] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370633] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370651] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370661] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-24 >[2018-08-18 10:05:05.370689] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370714] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370723] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370745] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370767] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370767] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-25 >[2018-08-18 10:05:05.370789] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370815] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370829] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370843] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-26 >[2018-08-18 10:05:05.370876] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370879] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370872] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370916] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370934] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-27 >[2018-08-18 10:05:05.370970] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.370992] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371000] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371028] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371054] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-28 >[2018-08-18 10:05:05.371056] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371078] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371139] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371152] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371165] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371185] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371214] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371215] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-30 >[2018-08-18 10:05:05.371233] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371257] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371272] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371281] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-31 >[2018-08-18 10:05:05.371323] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371361] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371368] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371382] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371391] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-32 >[2018-08-18 10:05:05.371460] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371453] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371457] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371507] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371513] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371524] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-33 >[2018-08-18 10:05:05.371540] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371542] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371571] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371598] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371588] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371608] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371633] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-34 >[2018-08-18 10:05:05.371636] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371637] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-29 >[2018-08-18 10:05:05.371633] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371681] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371697] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a) from subvol vol0-replicate-35 >[2018-08-18 10:05:05.371709] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-29 cached_subvol null >[2018-08-18 10:05:05.371722] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a >[2018-08-18 10:05:05.371732] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371742] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371758] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371768] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.371789] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34019200, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.372102] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-0': avail_percent is: 96.00 and avail_space is: 7683146395648 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372132] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-2': avail_percent is: 96.00 and avail_space is: 7682652368896 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372181] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-3': avail_percent is: 96.00 and avail_space is: 7679958376448 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372227] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-1': avail_percent is: 95.00 and avail_space is: 7649950310400 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372260] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-6': avail_percent is: 94.00 and avail_space is: 7572583333888 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372291] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-5': avail_percent is: 95.00 and avail_space is: 7674803892224 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372323] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-4': avail_percent is: 96.00 and avail_space is: 7680306479104 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372364] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-7': avail_percent is: 95.00 and avail_space is: 7602560430080 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372388] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-8': avail_percent is: 95.00 and avail_space is: 7674537177088 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372409] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-9': avail_percent is: 95.00 and avail_space is: 7675655356416 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372439] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-10': avail_percent is: 94.00 and avail_space is: 7587969724416 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372468] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-11': avail_percent is: 95.00 and avail_space is: 7666756444160 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372533] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-12': avail_percent is: 95.00 and avail_space is: 7662852456448 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372573] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-13': avail_percent is: 95.00 and avail_space is: 7672171716608 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372598] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-14': avail_percent is: 95.00 and avail_space is: 7671006838784 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372625] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-15': avail_percent is: 95.00 and avail_space is: 7657355005952 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372649] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-17': avail_percent is: 95.00 and avail_space is: 7670377443328 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372671] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-16': avail_percent is: 96.00 and avail_space is: 7681130377216 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372707] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-18': avail_percent is: 95.00 and avail_space is: 7620654678016 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372750] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-19': avail_percent is: 95.00 and avail_space is: 7661022814208 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372796] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-20': avail_percent is: 95.00 and avail_space is: 7611642155008 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372827] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-21': avail_percent is: 95.00 and avail_space is: 7663950176256 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372845] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-22': avail_percent is: 95.00 and avail_space is: 7668892258304 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372865] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-23': avail_percent is: 95.00 and avail_space is: 7670234238976 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372909] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-24': avail_percent is: 95.00 and avail_space is: 7665744367616 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372936] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-25': avail_percent is: 95.00 and avail_space is: 7650961227776 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372980] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-27': avail_percent is: 95.00 and avail_space is: 7652756992000 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.372999] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-26': avail_percent is: 95.00 and avail_space is: 7659003944960 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.373040] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-28': avail_percent is: 95.00 and avail_space is: 7652597972992 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.373065] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-29': avail_percent is: 95.00 and avail_space is: 7639883448320 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.373135] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-31': avail_percent is: 95.00 and avail_space is: 7678388396032 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.373162] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-30': avail_percent is: 95.00 and avail_space is: 7677991510016 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.373192] D [MSGID: 0] [dht-common.c:6923:dht_create_wind_to_avail_subvol] 0-vol0-dht: creating /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a on vol0-replicate-29 >[2018-08-18 10:05:05.373200] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-33': avail_percent is: 95.00 and avail_space is: 7679374594048 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.373231] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-32': avail_percent is: 95.00 and avail_space is: 7664158932992 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.373256] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-34': avail_percent is: 95.00 and avail_space is: 7668156858368 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.373281] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-35': avail_percent is: 95.00 and avail_space is: 7666712231936 and avail_inodes is: 99.00 >[2018-08-18 10:05:05.373370] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-29: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.375389] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 10:05:05.375593] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 10:05:05.375760] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 10:05:05.375812] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 10:05:05.376053] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-22: lk op is for a transaction >[2018-08-18 10:05:05.376296] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 10:05:05.377222] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 10:05:05.377250] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 10:05:05.377268] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 10:05:05.377331] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 10:05:05.377555] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 10:05:05.377828] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 10:05:05.377869] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 10:05:05.378059] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 10:05:05.378891] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 10:05:05.379064] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 10:05:05.379875] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 10:05:05.379997] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:05.380138] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a on vol0-replicate-34 >[2018-08-18 10:05:05.380227] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 10:05:05.380392] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.380403] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.380417] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.380441] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.380452] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a with op_ret -1 [No such file or directory] >[2018-08-18 10:05:05.380463] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a missing on subvol vol0-replicate-34 >[2018-08-18 10:05:05.380485] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:05.380677] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:05.380739] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.380742] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.380749] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.380797] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.380798] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.380827] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-0 >[2018-08-18 10:05:05.380834] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.380813] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.380870] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.380880] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.380892] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-1 >[2018-08-18 10:05:05.380912] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.380930] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.380952] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.380964] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.380963] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.380988] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-2 >[2018-08-18 10:05:05.380987] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381024] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381035] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381051] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-3 >[2018-08-18 10:05:05.381066] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381079] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381097] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381112] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-4 >[2018-08-18 10:05:05.381113] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381115] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381149] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381168] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381180] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-5 >[2018-08-18 10:05:05.381181] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381214] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381217] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381248] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381268] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-6 >[2018-08-18 10:05:05.381305] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381321] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381339] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381363] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381378] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-7 >[2018-08-18 10:05:05.381423] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381437] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381446] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381450] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381482] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381501] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381507] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-8 >[2018-08-18 10:05:05.381508] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381546] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381578] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381584] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381578] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-9 >[2018-08-18 10:05:05.381626] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381657] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381672] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-10 >[2018-08-18 10:05:05.381688] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381722] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381732] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381760] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381765] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-20: lk op is for a transaction >[2018-08-18 10:05:05.381772] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381775] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381791] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-11 >[2018-08-18 10:05:05.381834] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381842] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381857] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381866] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381878] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381888] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381904] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381934] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381906] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-12 >[2018-08-18 10:05:05.381928] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-13 >[2018-08-18 10:05:05.381968] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381982] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.381984] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382017] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382021] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382049] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382067] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382074] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-14 >[2018-08-18 10:05:05.382077] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382082] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-15 >[2018-08-18 10:05:05.382106] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382124] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382133] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-16 >[2018-08-18 10:05:05.382139] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382175] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382176] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382207] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382218] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-17 >[2018-08-18 10:05:05.382231] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382251] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382280] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382304] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382319] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-18 >[2018-08-18 10:05:05.382323] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382335] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382363] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382380] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382389] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-19 >[2018-08-18 10:05:05.382405] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382429] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382440] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382495] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382500] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382520] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382520] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382521] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-20 >[2018-08-18 10:05:05.382563] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382573] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382578] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-21 >[2018-08-18 10:05:05.382607] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:05.382609] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382609] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382638] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382647] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-22 >[2018-08-18 10:05:05.382690] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382713] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382734] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382749] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382766] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-23 >[2018-08-18 10:05:05.382805] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382820] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382821] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382850] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382861] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-24 >[2018-08-18 10:05:05.382877] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382886] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382906] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382921] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382931] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-25 >[2018-08-18 10:05:05.382960] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382973] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.382966] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383007] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383018] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-26 >[2018-08-18 10:05:05.383023] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383063] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383066] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383098] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383099] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383109] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383136] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-27 >[2018-08-18 10:05:05.383138] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383165] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383166] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383166] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383184] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-28 >[2018-08-18 10:05:05.383196] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383218] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383220] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383229] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-29 >[2018-08-18 10:05:05.383256] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383262] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383284] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383292] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383309] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383315] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-30 >[2018-08-18 10:05:05.383325] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383345] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383358] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-31 >[2018-08-18 10:05:05.383420] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383421] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383449] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383463] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383474] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-32 >[2018-08-18 10:05:05.383480] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383504] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383531] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383549] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383556] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-33 >[2018-08-18 10:05:05.383579] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383584] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383609] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383627] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383635] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-34 >[2018-08-18 10:05:05.383654] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383680] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383694] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383727] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383743] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a) from subvol vol0-replicate-35 >[2018-08-18 10:05:05.383759] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-34 cached_subvol null >[2018-08-18 10:05:05.383769] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a >[2018-08-18 10:05:05.383780] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383791] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383805] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383816] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383827] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.383921] D [MSGID: 0] [dht-common.c:6923:dht_create_wind_to_avail_subvol] 0-vol0-dht: creating /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a on vol0-replicate-34 >[2018-08-18 10:05:05.384107] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-34: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.385969] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 10:05:05.387508] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 10:05:05.387873] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:05.388060] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:05.394741] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7f3c34065ed0) as it exceeds the maximum available buffer size >[2018-08-18 10:05:05.395487] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.395507] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=30e78aeb-d190-4a89-bf3b-03616bdf8153, gen=0): picking the request for winding >[2018-08-18 10:05:05.395518] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=30e78aeb-d190-4a89-bf3b-03616bdf8153, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:05.395571] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-1: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.395582] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 10:05:05.395601] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7f3c34057ad0) allocated with standard calloc() >[2018-08-18 10:05:05.395598] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-30: lk op is for a transaction >[2018-08-18 10:05:05.396361] D [MSGID: 0] [dht-selfheal.c:1879:dht_selfheal_layout_new_directory] 0-vol0-dht: chunk size = 0xffffffff / 274638528 = 15.638619 >[2018-08-18 10:05:05.396382] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-18 >[2018-08-18 10:05:05.396390] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-19 >[2018-08-18 10:05:05.396396] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-2 >[2018-08-18 10:05:05.396402] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-20 >[2018-08-18 10:05:05.396408] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-21 >[2018-08-18 10:05:05.396413] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-22 >[2018-08-18 10:05:05.396419] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-23 >[2018-08-18 10:05:05.396424] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-24 >[2018-08-18 10:05:05.396429] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-25 >[2018-08-18 10:05:05.396434] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-26 >[2018-08-18 10:05:05.396445] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-27 >[2018-08-18 10:05:05.396450] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-28 >[2018-08-18 10:05:05.396456] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-29 >[2018-08-18 10:05:05.396461] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-3 >[2018-08-18 10:05:05.396466] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-30 >[2018-08-18 10:05:05.396471] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-31 >[2018-08-18 10:05:05.396476] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-32 >[2018-08-18 10:05:05.396481] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-33 >[2018-08-18 10:05:05.396502] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-34 >[2018-08-18 10:05:05.396507] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-35 >[2018-08-18 10:05:05.396512] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-4 >[2018-08-18 10:05:05.396516] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-5 >[2018-08-18 10:05:05.396521] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-6 >[2018-08-18 10:05:05.396526] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-7 >[2018-08-18 10:05:05.396531] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-8 >[2018-08-18 10:05:05.396535] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-9 >[2018-08-18 10:05:05.396540] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-0 >[2018-08-18 10:05:05.396546] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-1 >[2018-08-18 10:05:05.396552] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-10 >[2018-08-18 10:05:05.396557] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-11 >[2018-08-18 10:05:05.396562] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-12 >[2018-08-18 10:05:05.396567] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-13 >[2018-08-18 10:05:05.396571] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-14 >[2018-08-18 10:05:05.396576] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-15 >[2018-08-18 10:05:05.396581] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-16 >[2018-08-18 10:05:05.396588] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-17 >[2018-08-18 10:05:05.396652] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-23: lk op is for a transaction >[2018-08-18 10:05:05.396831] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-45: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.396853] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c600039d0, vol0-client-45 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.396902] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-46: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.396922] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c600039d0, vol0-client-46 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.397028] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-47: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:05.397071] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c600039d0, vol0-client-47 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:05.400884] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-29 >[2018-08-18 10:05:05.400988] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:05.401735] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-34 >[2018-08-18 10:05:05.401790] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 10:05:05.406634] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.406660] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=1, gfid=30e78aeb-d190-4a89-bf3b-03616bdf8153. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=30e78aeb-d190-4a89-bf3b-03616bdf8153, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:05.411941] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=30e78aeb-d190-4a89-bf3b-03616bdf8153, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:05.412094] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=30e78aeb-d190-4a89-bf3b-03616bdf8153, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.412234] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.412252] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=30e78aeb-d190-4a89-bf3b-03616bdf8153, gen=1): picking the request for winding >[2018-08-18 10:05:05.412467] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=30e78aeb-d190-4a89-bf3b-03616bdf8153, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:05.412659] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 10:05:05.412853] D [MSGID: 109036] [dht-common.c:9587:dht_log_new_layout_for_dir_selfheal] 0-vol0-dht: Setting layout of /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m with [Subvol_name: vol0-replicate-0, Err: -1 , Start: 3101920822 , Stop: 3221225468 , Hash: 1 ], [Subvol_name: vol0-replicate-1, Err: -1 , Start: 3221225469 , Stop: 3340530115 , Hash: 1 ], [Subvol_name: vol0-replicate-10, Err: -1 , Start: 3340530116 , Stop: 3459834762 , Hash: 1 ], [Subvol_name: vol0-replicate-11, Err: -1 , Start: 3459834763 , Stop: 3579139409 , Hash: 1 ], [Subvol_name: vol0-replicate-12, Err: -1 , Start: 3579139410 , Stop: 3698444056 , Hash: 1 ], [Subvol_name: vol0-replicate-13, Err: -1 , Start: 3698444057 , Stop: 3817748703 , Hash: 1 ], [Subvol_name: vol0-replicate-14, Err: -1 , Start: 3817748704 , Stop: 3937053350 , Hash: 1 ], [Subvol_name: vol0-replicate-15, Err: -1 , Start: 3937053351 , Stop: 4056357997 , Hash: 1 ], [Subvol_name: vol0-replicate-16, Err: -1 , Start: 4056357998 , Stop: 4175662644 , Hash: 1 ], [Subvol_name: vol0-replicate-17, Err: -1 , Start: 4175662645 , Stop: 4294967295 , Hash: 1 ], [Subvol_name: vol0-replicate-18, Err: -1 , Start: 0 , Stop: 119304646 , Hash: 1 ], [Subvol_name: vol0-replicate-19, Err: -1 , Start: 119304647 , Stop: 238609293 , Hash: 1 ], [Subvol_name: vol0-replicate-2, Err: -1 , Start: 238609294 , Stop: 357913940 , Hash: 1 ], [Subvol_name: vol0-replicate-20, Err: -1 , Start: 357913941 , Stop: 477218587 , Hash: 1 ], [Subvol_name: vol0-replicate-21, Err: -1 , Start: 477218588 , Stop: 596523234 , Hash: 1 ], [Subvol_name: vol0-replicate-22, Err: -1 , Start: 596523235 , Stop: 715827881 , Hash: 1 ], [Subvol_name: vol0-replicate-23, Err: -1 , Start: 715827882 , Stop: 835132528 , Hash: 1 ], [Subvol_name: vol0-replicate-24, Err: -1 , Start: 835132529 , Stop: 954437175 , Hash: 1 ], [Subvol_name: vol0-replicate-25, Err: -1 , Start: 954437176 , Stop: 1073741822 , Hash: 1 ], [Subvol_name: vol0-replicate-26, Err: -1 , Start: 1073741823 , Stop: 1193046469 , Hash: 1 ], [Subvol_name: vol0-replicate-27, Err: -1 , Start: 1193046470 , Stop: 1312351116 , Hash: 1 ], [Subvol_name: vol0-replicate-28, Err: -1 , Start: 1312351117 , Stop: 1431655763 , Hash: 1 ], [Subvol_name: vol0-replicate-29, Err: -1 , Start: 1431655764 , Stop: 1550960410 , Hash: 1 ], [Subvol_name: vol0-replicate-3, Err: -1 , Start: 1550960411 , Stop: 1670265057 , Hash: 1 ], [Subvol_name: vol0-replicate-30, Err: -1 , Start: 1670265058 , Stop: 1789569704 , Hash: 1 ], [Subvol_name: vol0-replicate-31, Err: -1 , Start: 1789569705 , Stop: 1908874351 , Hash: 1 ], [Subvol_name: vol0-replicate-32, Err: -1 , Start: 1908874352 , Stop: 2028178998 , Hash: 1 ], [Subvol_name: vol0-replicate-33, Err: -1 , Start: 2028178999 , Stop: 2147483645 , Hash: 1 ], [Subvol_name: vol0-replicate-34, Err: -1 , Start: 2147483646 , Stop: 2266788292 , Hash: 1 ], [Subvol_name: vol0-replicate-35, Err: -1 , Start: 2266788293 , Stop: 2386092939 , Hash: 1 ], [Subvol_name: vol0-replicate-4, Err: -1 , Start: 2386092940 , Stop: 2505397586 , Hash: 1 ], [Subvol_name: vol0-replicate-5, Err: -1 , Start: 2505397587 , Stop: 2624702233 , Hash: 1 ], [Subvol_name: vol0-replicate-6, Err: -1 , Start: 2624702234 , Stop: 2744006880 , Hash: 1 ], [Subvol_name: vol0-replicate-7, Err: -1 , Start: 2744006881 , Stop: 2863311527 , Hash: 1 ], [Subvol_name: vol0-replicate-8, Err: -1 , Start: 2863311528 , Stop: 2982616174 , Hash: 1 ], [Subvol_name: vol0-replicate-9, Err: -1 , Start: 2982616175 , Stop: 3101920821 , Hash: 1 ], >[2018-08-18 10:05:05.412875] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7f3c34065ed0) allocated with standard calloc() >[2018-08-18 10:05:05.413140] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-0: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.413221] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-1: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.413338] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.413393] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 10:05:05.413450] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.413491] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 10:05:05.413577] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-12: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.413602] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 10:05:05.413649] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 10:05:05.413735] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.413815] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.413816] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 10:05:05.413846] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004 on vol0-replicate-6 >[2018-08-18 10:05:05.413977] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-15: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.414016] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 10:05:05.414029] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:05.414067] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.414113] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-17: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.414162] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414175] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414213] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 10:05:05.414216] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414235] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-18: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.414252] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414274] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004 with op_ret -1 [No such file or directory] >[2018-08-18 10:05:05.414292] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004 missing on subvol vol0-replicate-6 >[2018-08-18 10:05:05.414302] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:05.414323] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:05.414347] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 10:05:05.414349] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-19: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.414386] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-2: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.414479] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 10:05:05.414500] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-20: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.414562] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.414586] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414591] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414622] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 10:05:05.414626] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414640] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414648] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-0 >[2018-08-18 10:05:05.414651] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 10:05:05.414687] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414697] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414699] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-22: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.414713] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414717] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-20: lk op is for a transaction >[2018-08-18 10:05:05.414735] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414771] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414761] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414788] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414787] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:05.414805] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414813] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-1 >[2018-08-18 10:05:05.414813] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414836] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-2 >[2018-08-18 10:05:05.414837] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-23: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.414840] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414855] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.414841] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414887] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414908] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-22: lk op is for a transaction >[2018-08-18 10:05:05.414893] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414921] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-25: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.414920] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414930] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-3 >[2018-08-18 10:05:05.414921] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414962] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414976] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414984] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.414987] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415009] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-4 >[2018-08-18 10:05:05.415012] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-26: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.415038] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415043] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415060] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415061] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-5 >[2018-08-18 10:05:05.415091] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 10:05:05.415091] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-27: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.415128] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415132] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 10:05:05.415154] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-23: lk op is for a transaction >[2018-08-18 10:05:05.415177] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415191] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415208] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415208] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-28: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.415220] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-7 >[2018-08-18 10:05:05.415220] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415233] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415264] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415276] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-29: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.415288] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415262] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-26: lk op is for a transaction >[2018-08-18 10:05:05.415304] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-8 >[2018-08-18 10:05:05.415344] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415353] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-27: lk op is for a transaction >[2018-08-18 10:05:05.415345] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415356] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415378] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415385] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-9 >[2018-08-18 10:05:05.415418] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415411] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-3: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.415421] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415454] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 10:05:05.415459] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-30: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.415484] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415477] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415514] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415533] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-10 >[2018-08-18 10:05:05.415533] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:05.415542] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415559] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415570] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415574] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415611] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415623] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415635] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-11 >[2018-08-18 10:05:05.415643] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415656] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415646] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.415658] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415665] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-32: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.415686] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415702] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415699] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415699] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-12 >[2018-08-18 10:05:05.415724] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415743] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-13 >[2018-08-18 10:05:05.415725] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 10:05:05.415744] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415731] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-30: lk op is for a transaction >[2018-08-18 10:05:05.415767] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-33: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.415779] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415796] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-14 >[2018-08-18 10:05:05.415797] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415814] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415824] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415837] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-34: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.415848] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415858] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-15 >[2018-08-18 10:05:05.415924] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 10:05:05.415923] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415961] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:05.415927] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415974] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.415992] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416013] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416018] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416029] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 10:05:05.416055] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-16 >[2018-08-18 10:05:05.416061] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416066] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.416072] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 10:05:05.416084] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416086] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416098] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-17 >[2018-08-18 10:05:05.416118] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-5: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.416123] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416152] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416154] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416167] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416178] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416181] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416224] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-18 >[2018-08-18 10:05:05.416228] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416230] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416239] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-6: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.416228] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416258] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-19 >[2018-08-18 10:05:05.416272] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416293] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416302] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416306] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-20 >[2018-08-18 10:05:05.416336] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.416316] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416317] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416358] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416374] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416384] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 10:05:05.416387] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-21 >[2018-08-18 10:05:05.416391] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416423] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-8: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.416423] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416433] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-35: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.416446] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416443] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 10:05:05.416451] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416458] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-22 >[2018-08-18 10:05:05.416485] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416479] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 10:05:05.416495] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416524] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416533] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-23 >[2018-08-18 10:05:05.416551] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 10:05:05.416558] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416558] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416571] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-9: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.416579] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416591] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416598] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-24 >[2018-08-18 10:05:05.416601] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416630] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416632] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 10:05:05.416638] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416655] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416662] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-25 >[2018-08-18 10:05:05.416689] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 10:05:05.416700] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416737] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416779] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416793] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416799] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416807] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416825] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-26 >[2018-08-18 10:05:05.416825] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416839] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416857] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416858] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416873] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-27 >[2018-08-18 10:05:05.416884] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416904] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416913] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-28 >[2018-08-18 10:05:05.416918] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416922] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416948] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416959] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416966] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-29 >[2018-08-18 10:05:05.416978] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416990] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.416999] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417010] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 10:05:05.417020] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417055] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-30 >[2018-08-18 10:05:05.417057] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417079] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417080] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417129] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417141] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417142] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-31 >[2018-08-18 10:05:05.417149] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417142] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417195] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417205] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-32 >[2018-08-18 10:05:05.417210] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417216] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417253] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417281] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417294] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-33 >[2018-08-18 10:05:05.417315] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417334] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417348] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417361] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417377] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417358] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-6 >[2018-08-18 10:05:05.417399] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417409] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-34 >[2018-08-18 10:05:05.417449] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417450] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417492] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417513] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417527] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-35 >[2018-08-18 10:05:05.417543] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-6 cached_subvol null >[2018-08-18 10:05:05.417554] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004 >[2018-08-18 10:05:05.417566] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417578] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417592] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417605] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.417618] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.418267] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m returned with op_ret 0 >[2018-08-18 10:05:05.418345] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a on vol0-replicate-16 >[2018-08-18 10:05:05.418616] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.418666] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.418693] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.418720] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.418737] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a with op_ret -1 [No such file or directory] >[2018-08-18 10:05:05.418759] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a missing on subvol vol0-replicate-16 >[2018-08-18 10:05:05.418773] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:05.419038] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419083] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419114] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419144] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419160] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-0 >[2018-08-18 10:05:05.419192] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419196] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419234] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419262] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419295] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-1 >[2018-08-18 10:05:05.419308] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419329] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419346] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419366] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419377] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-2 >[2018-08-18 10:05:05.419406] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419434] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419464] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419481] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419490] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-3 >[2018-08-18 10:05:05.419499] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419570] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419572] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419612] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419630] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-4 >[2018-08-18 10:05:05.419650] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419654] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419695] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419712] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419712] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419728] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419752] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-5 >[2018-08-18 10:05:05.419770] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419805] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419808] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419816] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-6 >[2018-08-18 10:05:05.419816] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419810] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419862] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419877] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-7 >[2018-08-18 10:05:05.419945] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419967] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.419977] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420008] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420013] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420042] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-8 >[2018-08-18 10:05:05.420045] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420071] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420093] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420106] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-9 >[2018-08-18 10:05:05.420153] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420167] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420205] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420231] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420225] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420226] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420267] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-10 >[2018-08-18 10:05:05.420269] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420288] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420296] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420302] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-11 >[2018-08-18 10:05:05.420320] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420364] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420383] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420394] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-12 >[2018-08-18 10:05:05.420445] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420447] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420462] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420483] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420491] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-13 >[2018-08-18 10:05:05.420534] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420570] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420571] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420609] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420618] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-14 >[2018-08-18 10:05:05.420621] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420651] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420665] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420678] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420685] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-15 >[2018-08-18 10:05:05.420786] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420842] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420849] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420861] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420976] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420997] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.420999] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421038] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421044] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421041] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421078] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421088] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-18 >[2018-08-18 10:05:05.421098] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-17 >[2018-08-18 10:05:05.421094] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421103] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421131] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421143] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421144] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421161] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421166] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-19 >[2018-08-18 10:05:05.421194] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421211] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-20 >[2018-08-18 10:05:05.421235] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421249] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421272] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421304] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421321] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-21 >[2018-08-18 10:05:05.421374] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421409] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421452] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421469] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421479] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-22 >[2018-08-18 10:05:05.421482] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421494] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421525] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421542] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421546] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421565] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-23 >[2018-08-18 10:05:05.421607] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421645] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421642] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.421652] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421661] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421673] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 10:05:05.421677] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m returned with op_ret 0" repeated 35 times between [2018-08-18 10:05:05.418267] and [2018-08-18 10:05:05.421128] >[2018-08-18 10:05:05.421671] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=30e78aeb-d190-4a89-bf3b-03616bdf8153, gen=0): picking the request for winding >[2018-08-18 10:05:05.421699] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421700] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421718] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-24 >[2018-08-18 10:05:05.421724] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a at vol0-replicate-1 >[2018-08-18 10:05:05.421725] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-25 >[2018-08-18 10:05:05.421736] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421761] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421790] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421806] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421813] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-26 >[2018-08-18 10:05:05.421815] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421850] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421868] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421917] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421923] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421939] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-27 >[2018-08-18 10:05:05.421956] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421983] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.421999] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422009] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-28 >[2018-08-18 10:05:05.422023] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422011] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=30e78aeb-d190-4a89-bf3b-03616bdf8153, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.422050] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422063] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a returned with op_ret 0 >[2018-08-18 10:05:05.422068] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422094] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422106] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-29 >[2018-08-18 10:05:05.422133] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422157] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422158] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422187] D [md-cache.c:503:mdc_inode_iatt_set_validate] (-->/usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3cd0) [0x7f3d02157cd0] -->/usr/lib64/glusterfs/3.10.12/xlator/performance/md-cache.so(+0xa802) [0x7f3d01d3e802] -->/usr/lib64/glusterfs/3.10.12/xlator/performance/md-cache.so(+0x66cb) [0x7f3d01d3a6cb] ) 0-vol0-md-cache: discarding the iatt validate request(ctime_nsec) [Invalid argument] >[2018-08-18 10:05:05.422197] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422208] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-30 >[2018-08-18 10:05:05.422234] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422256] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422276] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422309] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422323] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-31 >[2018-08-18 10:05:05.422350] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422356] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004 on vol0-replicate-6 >[2018-08-18 10:05:05.422384] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422397] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422417] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422424] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422432] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-32 >[2018-08-18 10:05:05.422442] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422433] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422443] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422445] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422464] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-16 >[2018-08-18 10:05:05.422512] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422475] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422518] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422533] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422529] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422540] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422520] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422586] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-33 >[2018-08-18 10:05:05.422587] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422592] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422622] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-34 >[2018-08-18 10:05:05.422633] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422631] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a) from subvol vol0-replicate-35 >[2018-08-18 10:05:05.422650] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-16 cached_subvol null >[2018-08-18 10:05:05.422642] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422652] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422659] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a >[2018-08-18 10:05:05.422671] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422671] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422685] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004 with op_ret -1 [No such file or directory] >[2018-08-18 10:05:05.422685] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422694] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004 missing on subvol vol0-replicate-6 >[2018-08-18 10:05:05.422696] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422701] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:05.422705] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422725] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405dc20, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422869] D [MSGID: 0] [dht-common.c:6923:dht_create_wind_to_avail_subvol] 0-vol0-dht: creating /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a on vol0-replicate-16 >[2018-08-18 10:05:05.422957] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422966] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.422967] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423003] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423005] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423028] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-0 >[2018-08-18 10:05:05.423045] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423039] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423036] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423078] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423083] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423080] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423087] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423102] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-16: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.423110] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423107] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-1 >[2018-08-18 10:05:05.423136] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423160] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423162] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-2 >[2018-08-18 10:05:05.423167] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423177] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423169] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423204] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423206] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423217] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-3 >[2018-08-18 10:05:05.423222] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-4 >[2018-08-18 10:05:05.423267] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423273] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423300] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423322] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423334] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-5 >[2018-08-18 10:05:05.423341] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423357] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423462] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423462] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423469] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423499] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423503] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423501] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423532] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-7 >[2018-08-18 10:05:05.423530] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423558] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423567] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-8 >[2018-08-18 10:05:05.423581] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423605] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423639] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423671] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423680] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423683] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423705] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-9 >[2018-08-18 10:05:05.423713] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423751] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423777] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423777] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-10 >[2018-08-18 10:05:05.423779] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423788] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423834] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423848] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423835] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423877] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-11 >[2018-08-18 10:05:05.423879] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423882] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423902] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423882] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423920] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423925] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-12 >[2018-08-18 10:05:05.423953] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423960] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.423973] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-13 >[2018-08-18 10:05:05.423986] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424019] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424041] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424062] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424078] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-14 >[2018-08-18 10:05:05.424071] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424087] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424118] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424136] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424120] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424177] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424191] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-15 >[2018-08-18 10:05:05.424200] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424219] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-16 >[2018-08-18 10:05:05.424221] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424242] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424245] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424258] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424276] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424297] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-17 >[2018-08-18 10:05:05.424318] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424337] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424337] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424338] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424365] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424366] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424381] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-18 >[2018-08-18 10:05:05.424394] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424408] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-19 >[2018-08-18 10:05:05.424409] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424417] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424418] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424448] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424462] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-20 >[2018-08-18 10:05:05.424485] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424493] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424551] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424581] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424582] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424585] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424614] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-21 >[2018-08-18 10:05:05.424629] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424644] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424652] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-22 >[2018-08-18 10:05:05.424658] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424664] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424726] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424743] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424747] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424744] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424772] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-23 >[2018-08-18 10:05:05.424775] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424798] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424808] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-24 >[2018-08-18 10:05:05.424809] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424832] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424834] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424855] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424864] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-25 >[2018-08-18 10:05:05.424904] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424898] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424933] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424940] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424963] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424978] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.424987] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-27 >[2018-08-18 10:05:05.424984] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425019] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425051] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425049] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425084] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-26 >[2018-08-18 10:05:05.425135] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425158] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425160] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425178] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-28 >[2018-08-18 10:05:05.425181] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425186] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425185] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425224] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425232] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425241] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425253] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425253] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425276] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-29 >[2018-08-18 10:05:05.425294] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425310] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425323] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425343] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425353] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425349] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425362] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425359] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-31 >[2018-08-18 10:05:05.425349] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425393] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-32 >[2018-08-18 10:05:05.425408] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425430] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425455] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425460] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425476] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425456] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425505] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425490] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425488] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-30 >[2018-08-18 10:05:05.425501] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-6 >[2018-08-18 10:05:05.425512] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-33 >[2018-08-18 10:05:05.425499] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425548] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425562] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425565] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425596] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-34 >[2018-08-18 10:05:05.425598] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425618] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004) from subvol vol0-replicate-35 >[2018-08-18 10:05:05.425629] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-6 cached_subvol null >[2018-08-18 10:05:05.425638] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004 >[2018-08-18 10:05:05.425646] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425655] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425665] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425674] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425683] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c041ba0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.425835] I [MSGID: 109066] [dht-rename.c:1610:dht_rename] 0-vol0-dht: renaming /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004_2d3760c6a2ce11e8850b246e9631871a (hash=vol0-replicate-1/cache=vol0-replicate-1) => /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004 (hash=vol0-replicate-6/cache=<nul>) >[2018-08-18 10:05:05.426600] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-6: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.426956] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 10:05:05.427063] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-1: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.427395] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 10:05:05.427530] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-6: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.428110] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 10:05:05.428207] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-1: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.428520] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 10:05:05.428574] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-6: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.428758] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 10:05:05.430410] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.430430] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=30e78aeb-d190-4a89-bf3b-03616bdf8153, gen=0): picking the request for winding >[2018-08-18 10:05:05.430453] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004 at vol0-replicate-1 >[2018-08-18 10:05:05.430649] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=30e78aeb-d190-4a89-bf3b-03616bdf8153, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.430769] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004 returned with op_ret 0 >[2018-08-18 10:05:05.430987] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-1: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.431225] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 10:05:05.431563] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.431580] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=30e78aeb-d190-4a89-bf3b-03616bdf8153, gen=0): picking the request for winding >[2018-08-18 10:05:05.431601] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004 at vol0-replicate-1 >[2018-08-18 10:05:05.431776] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=30e78aeb-d190-4a89-bf3b-03616bdf8153, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.431912] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004 returned with op_ret 0 >[2018-08-18 10:05:05.432142] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-1: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.432339] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 10:05:05.432511] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.432528] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=30e78aeb-d190-4a89-bf3b-03616bdf8153, gen=0): picking the request for winding >[2018-08-18 10:05:05.432548] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004 at vol0-replicate-1 >[2018-08-18 10:05:05.432719] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=30e78aeb-d190-4a89-bf3b-03616bdf8153, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.432824] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004 returned with op_ret 0 >[2018-08-18 10:05:05.433045] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-1: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.433253] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 10:05:05.433537] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.433554] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=30e78aeb-d190-4a89-bf3b-03616bdf8153, gen=0): picking the request for winding >[2018-08-18 10:05:05.433574] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004 at vol0-replicate-1 >[2018-08-18 10:05:05.433748] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=30e78aeb-d190-4a89-bf3b-03616bdf8153, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.433889] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/244a0f757c883c6fae718df8adeb6727_171632299417284990.m/0004 returned with op_ret 0 >[2018-08-18 10:05:05.434168] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-1: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.434387] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 10:05:05.434830] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-16 >[2018-08-18 10:05:05.434864] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:05.510482] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7f3c34024510) as it exceeds the maximum available buffer size >[2018-08-18 10:05:05.513317] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.513351] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8c5892ee-79c5-422e-a23b-b0b15f50f3f4, gen=0): picking the request for winding >[2018-08-18 10:05:05.513372] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8c5892ee-79c5-422e-a23b-b0b15f50f3f4, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:05.513712] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-29: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.517142] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7f3c2c012180) as it exceeds the maximum available buffer size >[2018-08-18 10:05:05.518563] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.518601] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=1a59df5f-819d-44a9-8627-99a29744138f, gen=0): picking the request for winding >[2018-08-18 10:05:05.518624] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=1a59df5f-819d-44a9-8627-99a29744138f, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:05.519049] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-34: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.530084] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7f3c3402a7f0) as it exceeds the maximum available buffer size >[2018-08-18 10:05:05.532449] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.532491] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=43698123-f668-4322-953f-f1a0f9a8eb33, gen=0): picking the request for winding >[2018-08-18 10:05:05.532511] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=43698123-f668-4322-953f-f1a0f9a8eb33, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:05.532963] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.534928] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8c5892ee-79c5-422e-a23b-b0b15f50f3f4, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:05.535080] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=8c5892ee-79c5-422e-a23b-b0b15f50f3f4, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.535206] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.544288] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=1a59df5f-819d-44a9-8627-99a29744138f, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:05.544427] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=1a59df5f-819d-44a9-8627-99a29744138f, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.544541] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.552531] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=43698123-f668-4322-953f-f1a0f9a8eb33, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:05.552695] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=43698123-f668-4322-953f-f1a0f9a8eb33, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.552836] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.637679] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7f3c3406e050) as it exceeds the maximum available buffer size >[2018-08-18 10:05:05.641282] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.641320] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=43698123-f668-4322-953f-f1a0f9a8eb33, gen=0): picking the request for winding >[2018-08-18 10:05:05.641341] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=43698123-f668-4322-953f-f1a0f9a8eb33, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:05.641424] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.641445] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:05.641491] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7f3c3402a7f0) allocated with standard calloc() >[2018-08-18 10:05:05.652412] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.652463] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=1, gfid=43698123-f668-4322-953f-f1a0f9a8eb33. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=43698123-f668-4322-953f-f1a0f9a8eb33, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:05.656785] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=43698123-f668-4322-953f-f1a0f9a8eb33, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:05.657012] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=43698123-f668-4322-953f-f1a0f9a8eb33, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.657230] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.657257] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=43698123-f668-4322-953f-f1a0f9a8eb33, gen=1): picking the request for winding >[2018-08-18 10:05:05.657686] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=43698123-f668-4322-953f-f1a0f9a8eb33, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:05.657814] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:05.658029] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7f3c3406e050) allocated with standard calloc() >[2018-08-18 10:05:05.659000] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002 on vol0-replicate-28 >[2018-08-18 10:05:05.659317] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.659440] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.659463] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.659509] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.659529] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002 with op_ret -1 [No such file or directory] >[2018-08-18 10:05:05.659546] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002 missing on subvol vol0-replicate-28 >[2018-08-18 10:05:05.659560] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:05.659953] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.659981] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660001] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660058] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660063] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660070] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660130] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-0 >[2018-08-18 10:05:05.660100] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660168] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660170] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660179] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660234] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-1 >[2018-08-18 10:05:05.660245] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660272] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660291] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660313] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660335] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-2 >[2018-08-18 10:05:05.660327] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660379] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660410] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660399] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-3 >[2018-08-18 10:05:05.660414] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660415] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660455] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660466] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-4 >[2018-08-18 10:05:05.660545] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660569] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660606] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660652] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660683] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-5 >[2018-08-18 10:05:05.660686] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660695] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660710] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660742] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660763] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-6 >[2018-08-18 10:05:05.660774] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660770] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660769] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660822] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660836] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660846] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-7 >[2018-08-18 10:05:05.660884] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660928] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660933] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660971] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660981] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.660981] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661037] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661045] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-8 >[2018-08-18 10:05:05.661046] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661074] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661084] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-9 >[2018-08-18 10:05:05.661115] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661142] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661193] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661200] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661202] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661194] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-10 >[2018-08-18 10:05:05.661242] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661264] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-11 >[2018-08-18 10:05:05.661267] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661306] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661333] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661355] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661368] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-12 >[2018-08-18 10:05:05.661429] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661447] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661477] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661489] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661497] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-13 >[2018-08-18 10:05:05.661514] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661529] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661547] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661562] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661573] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-14 >[2018-08-18 10:05:05.661642] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661648] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661684] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661701] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661711] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-15 >[2018-08-18 10:05:05.661769] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661791] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661804] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661806] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661844] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661840] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661869] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-16 >[2018-08-18 10:05:05.661876] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661876] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661906] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661927] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661926] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661956] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.661959] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-17 >[2018-08-18 10:05:05.661967] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-18 >[2018-08-18 10:05:05.661995] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662039] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662071] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662090] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662099] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-19 >[2018-08-18 10:05:05.662096] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662131] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662157] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662171] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662179] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-20 >[2018-08-18 10:05:05.662244] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662281] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662289] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662316] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662343] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-21 >[2018-08-18 10:05:05.662392] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662399] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662421] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662451] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662457] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662467] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-22 >[2018-08-18 10:05:05.662496] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662500] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662523] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662539] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-23 >[2018-08-18 10:05:05.662568] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662571] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662624] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662665] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662669] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662685] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-24 >[2018-08-18 10:05:05.662712] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662715] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662748] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662758] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-25 >[2018-08-18 10:05:05.662783] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662828] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662851] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662870] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662889] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-26 >[2018-08-18 10:05:05.662889] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662891] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662901] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662954] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662963] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.662963] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-27 >[2018-08-18 10:05:05.662997] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663081] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663105] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663205] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663226] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663230] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663250] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-29 >[2018-08-18 10:05:05.663254] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663261] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663288] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663296] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-30 >[2018-08-18 10:05:05.663360] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663361] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663409] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663405] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663457] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663464] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663484] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-31 >[2018-08-18 10:05:05.663490] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663513] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663529] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-32 >[2018-08-18 10:05:05.663566] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663598] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663604] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663639] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663655] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-33 >[2018-08-18 10:05:05.663671] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663691] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663693] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663695] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663747] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663749] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663763] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663773] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-34 >[2018-08-18 10:05:05.663774] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663810] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-28 >[2018-08-18 10:05:05.663815] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663854] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663872] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-35 >[2018-08-18 10:05:05.663888] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-28 cached_subvol null >[2018-08-18 10:05:05.663900] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002 >[2018-08-18 10:05:05.663912] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x3434ce0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663926] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663942] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663958] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.663973] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x3434ce0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.664705] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m returned with op_ret 0 >[2018-08-18 10:05:05.668128] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7f3c34025030) as it exceeds the maximum available buffer size >[2018-08-18 10:05:05.668448] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.668480] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m returned with op_ret 0" repeated 35 times between [2018-08-18 10:05:05.664705] and [2018-08-18 10:05:05.667924] >[2018-08-18 10:05:05.668479] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=43698123-f668-4322-953f-f1a0f9a8eb33, gen=0): picking the request for winding >[2018-08-18 10:05:05.668551] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a at vol0-replicate-16 >[2018-08-18 10:05:05.668853] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=43698123-f668-4322-953f-f1a0f9a8eb33, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.668933] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a returned with op_ret 0 >[2018-08-18 10:05:05.669115] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002 on vol0-replicate-28 >[2018-08-18 10:05:05.669377] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.669431] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.669442] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.669472] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.669489] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002 with op_ret -1 [No such file or directory] >[2018-08-18 10:05:05.669505] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002 missing on subvol vol0-replicate-28 >[2018-08-18 10:05:05.669517] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:05.669822] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.669866] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.669884] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.669900] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.669941] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.669964] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.669972] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.669958] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.669986] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-0 >[2018-08-18 10:05:05.669992] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670011] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670013] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670038] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-1 >[2018-08-18 10:05:05.670055] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670078] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-2 >[2018-08-18 10:05:05.670104] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670107] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670108] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670138] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670148] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-3 >[2018-08-18 10:05:05.670209] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670234] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670258] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670266] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670295] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670313] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-4 >[2018-08-18 10:05:05.670317] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670366] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670369] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670414] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670409] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670435] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-5 >[2018-08-18 10:05:05.670452] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670455] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670497] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670505] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670507] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670511] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670545] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670575] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670570] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670586] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670598] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670575] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-6 >[2018-08-18 10:05:05.670633] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670639] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-7 >[2018-08-18 10:05:05.670650] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670678] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670681] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-8 >[2018-08-18 10:05:05.670688] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.670690] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-9 >[2018-08-18 10:05:05.670708] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8c5892ee-79c5-422e-a23b-b0b15f50f3f4, gen=0): picking the request for winding >[2018-08-18 10:05:05.670720] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8c5892ee-79c5-422e-a23b-b0b15f50f3f4, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:05.670717] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670736] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670768] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-29: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.670763] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670786] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:05.670758] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670809] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670823] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7f3c34024510) allocated with standard calloc() >[2018-08-18 10:05:05.670816] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670813] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670847] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-10 >[2018-08-18 10:05:05.670866] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670848] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670888] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670889] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-11 >[2018-08-18 10:05:05.670897] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670922] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.670932] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-12 >[2018-08-18 10:05:05.671028] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671025] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671041] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671092] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671107] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-13 >[2018-08-18 10:05:05.671140] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671183] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671205] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671234] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671246] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-14 >[2018-08-18 10:05:05.671278] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671280] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671316] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671316] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671329] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671371] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671354] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671379] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671371] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671405] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671388] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-15 >[2018-08-18 10:05:05.671427] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671451] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-16 >[2018-08-18 10:05:05.671448] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671448] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671455] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671496] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-17 >[2018-08-18 10:05:05.671503] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671523] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671536] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-18 >[2018-08-18 10:05:05.671602] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671682] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671696] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671683] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671731] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671727] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671760] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671752] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-19 >[2018-08-18 10:05:05.671804] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671806] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671815] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671817] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671858] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-20 >[2018-08-18 10:05:05.671860] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671873] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671881] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-21 >[2018-08-18 10:05:05.671885] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671885] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671947] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671956] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671959] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671969] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.671964] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-22 >[2018-08-18 10:05:05.672000] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672016] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-23 >[2018-08-18 10:05:05.672011] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672056] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672098] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672128] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672133] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672153] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-24 >[2018-08-18 10:05:05.672153] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672193] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672198] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672219] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672228] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672248] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-25 >[2018-08-18 10:05:05.672252] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672294] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672300] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672295] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672319] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-26 >[2018-08-18 10:05:05.672340] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672332] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672372] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672389] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-27 >[2018-08-18 10:05:05.672390] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672412] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672469] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672485] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672517] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672545] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-29 >[2018-08-18 10:05:05.672545] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672578] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672600] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672606] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672621] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672627] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672664] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672673] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672670] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672689] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-30 >[2018-08-18 10:05:05.672666] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672726] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672728] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-31 >[2018-08-18 10:05:05.672744] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672755] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672760] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-32 >[2018-08-18 10:05:05.672807] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672815] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672846] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672850] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672863] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672876] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672851] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672857] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672904] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-28 >[2018-08-18 10:05:05.672909] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672921] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-33 >[2018-08-18 10:05:05.672944] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672945] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-34 >[2018-08-18 10:05:05.672945] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.672978] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.673006] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.673017] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002) from subvol vol0-replicate-35 >[2018-08-18 10:05:05.673027] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-28 cached_subvol null >[2018-08-18 10:05:05.673056] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002 >[2018-08-18 10:05:05.673064] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.673073] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.673083] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.673093] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.673102] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c340035c0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.673244] I [MSGID: 109066] [dht-rename.c:1610:dht_rename] 0-vol0-dht: renaming /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002_2df4997aa2ce11e8a903246e9631871a (hash=vol0-replicate-16/cache=vol0-replicate-16) => /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002 (hash=vol0-replicate-28/cache=<nul>) >[2018-08-18 10:05:05.673317] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-11 >[2018-08-18 10:05:05.673622] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.673664] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.673655] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.673698] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.673730] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a with op_ret -1 [No such file or directory] >[2018-08-18 10:05:05.673751] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a missing on subvol vol0-replicate-11 >[2018-08-18 10:05:05.673764] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:05.674239] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674247] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-28: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.674249] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674278] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674294] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674298] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674279] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674330] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674355] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674360] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674365] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674369] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674378] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-1 >[2018-08-18 10:05:05.674398] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-0 >[2018-08-18 10:05:05.674386] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674412] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674441] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674462] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-2 >[2018-08-18 10:05:05.674483] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674487] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674508] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674526] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-3 >[2018-08-18 10:05:05.674535] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674544] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674567] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674583] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-4 >[2018-08-18 10:05:05.674583] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674642] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674645] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674674] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674686] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-5 >[2018-08-18 10:05:05.674687] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 10:05:05.674738] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674754] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674776] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674810] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-16: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.674811] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674835] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674835] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-6 >[2018-08-18 10:05:05.674854] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674935] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674933] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674970] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674975] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675000] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.674993] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-7 >[2018-08-18 10:05:05.674993] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675052] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675059] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675062] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675079] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675096] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675100] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-8 >[2018-08-18 10:05:05.675113] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-9 >[2018-08-18 10:05:05.675136] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675131] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675162] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675179] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-10 >[2018-08-18 10:05:05.675202] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:05.675237] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675300] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675402] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675413] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675417] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675423] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-28: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.675418] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675450] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675450] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675480] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675482] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675497] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-12 >[2018-08-18 10:05:05.675498] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-13 >[2018-08-18 10:05:05.675534] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675565] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675605] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675632] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675649] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-14 >[2018-08-18 10:05:05.675674] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675720] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675756] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675774] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675785] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-15 >[2018-08-18 10:05:05.675825] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675849] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675868] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675884] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675896] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-16 >[2018-08-18 10:05:05.675960] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675975] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.675978] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676022] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676061] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-17 >[2018-08-18 10:05:05.676092] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676101] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676097] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676134] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 10:05:05.676141] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676162] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-18 >[2018-08-18 10:05:05.676162] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676208] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676208] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676249] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676254] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676263] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-19 >[2018-08-18 10:05:05.676293] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676299] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676332] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676363] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676367] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-20 >[2018-08-18 10:05:05.676387] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-16: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.676407] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676411] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676444] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676455] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676466] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-21 >[2018-08-18 10:05:05.676486] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676489] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676522] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676539] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-22 >[2018-08-18 10:05:05.676564] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676564] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676601] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676626] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676641] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-23 >[2018-08-18 10:05:05.676684] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676704] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676733] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676754] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676768] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-24 >[2018-08-18 10:05:05.676818] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676864] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676892] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676908] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676917] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-25 >[2018-08-18 10:05:05.676936] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676954] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676939] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.676998] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677024] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-26 >[2018-08-18 10:05:05.677082] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677085] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677103] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677119] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677137] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677139] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677140] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677160] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-27 >[2018-08-18 10:05:05.677173] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677198] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-28 >[2018-08-18 10:05:05.677202] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677244] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677300] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677325] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677327] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677363] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-29 >[2018-08-18 10:05:05.677373] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677400] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677431] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677446] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-30 >[2018-08-18 10:05:05.677447] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677467] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677501] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677528] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677532] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:05.677527] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677559] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-31 >[2018-08-18 10:05:05.677544] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677576] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677596] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677606] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-32 >[2018-08-18 10:05:05.677629] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677642] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677653] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677673] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677693] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677720] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677735] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677720] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-11 >[2018-08-18 10:05:05.677736] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-28: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.677754] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677755] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-33 >[2018-08-18 10:05:05.677811] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677828] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677839] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-34 >[2018-08-18 10:05:05.677863] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677868] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677892] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677908] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677917] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a) from subvol vol0-replicate-35 >[2018-08-18 10:05:05.677925] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-11 cached_subvol null >[2018-08-18 10:05:05.677933] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a >[2018-08-18 10:05:05.677947] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677955] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677966] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677975] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.677984] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3402e2e0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.678131] D [MSGID: 0] [dht-common.c:6923:dht_create_wind_to_avail_subvol] 0-vol0-dht: creating /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a on vol0-replicate-11 >[2018-08-18 10:05:05.678157] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 10:05:05.678524] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-11: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.678612] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.678658] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=43698123-f668-4322-953f-f1a0f9a8eb33, gen=0): picking the request for winding >[2018-08-18 10:05:05.678701] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002 at vol0-replicate-16 >[2018-08-18 10:05:05.679005] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=43698123-f668-4322-953f-f1a0f9a8eb33, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.679197] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002 returned with op_ret 0 >[2018-08-18 10:05:05.679697] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.680107] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:05.680399] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.680429] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=43698123-f668-4322-953f-f1a0f9a8eb33, gen=0): picking the request for winding >[2018-08-18 10:05:05.680480] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002 at vol0-replicate-16 >[2018-08-18 10:05:05.680785] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=43698123-f668-4322-953f-f1a0f9a8eb33, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.681071] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002 returned with op_ret 0 >[2018-08-18 10:05:05.681511] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.681890] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:05.682228] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.682259] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=43698123-f668-4322-953f-f1a0f9a8eb33, gen=0): picking the request for winding >[2018-08-18 10:05:05.682291] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002 at vol0-replicate-16 >[2018-08-18 10:05:05.682603] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=43698123-f668-4322-953f-f1a0f9a8eb33, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.682792] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002 returned with op_ret 0 >[2018-08-18 10:05:05.683234] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.683635] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:05.683941] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7f3c2c0440d0) as it exceeds the maximum available buffer size >[2018-08-18 10:05:05.683967] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.683998] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=43698123-f668-4322-953f-f1a0f9a8eb33, gen=0): picking the request for winding >[2018-08-18 10:05:05.684029] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002 at vol0-replicate-16 >[2018-08-18 10:05:05.684333] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=43698123-f668-4322-953f-f1a0f9a8eb33, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.684537] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/8c6ddc8340be3104bee2743d556a2e68_171632304773416023.m/0002 returned with op_ret 0 >[2018-08-18 10:05:05.684813] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.685053] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:05.686003] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.686026] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=1, gfid=8c5892ee-79c5-422e-a23b-b0b15f50f3f4. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=8c5892ee-79c5-422e-a23b-b0b15f50f3f4, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:05.686723] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.686746] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=1a59df5f-819d-44a9-8627-99a29744138f, gen=0): picking the request for winding >[2018-08-18 10:05:05.686765] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=1a59df5f-819d-44a9-8627-99a29744138f, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:05.686816] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-34: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.686829] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 10:05:05.686852] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7f3c2c012180) allocated with standard calloc() >[2018-08-18 10:05:05.687085] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8c5892ee-79c5-422e-a23b-b0b15f50f3f4, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:05.687211] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=8c5892ee-79c5-422e-a23b-b0b15f50f3f4, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.687325] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.687341] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=8c5892ee-79c5-422e-a23b-b0b15f50f3f4, gen=1): picking the request for winding >[2018-08-18 10:05:05.687565] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=8c5892ee-79c5-422e-a23b-b0b15f50f3f4, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:05.687766] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:05.688020] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7f3c34025030) allocated with standard calloc() >[2018-08-18 10:05:05.689167] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001 on vol0-replicate-24 >[2018-08-18 10:05:05.689592] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.689605] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.689639] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.689656] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.689668] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001 with op_ret -1 [No such file or directory] >[2018-08-18 10:05:05.689678] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001 missing on subvol vol0-replicate-24 >[2018-08-18 10:05:05.689687] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:05.689969] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690010] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690053] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690074] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690108] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690091] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690113] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690115] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690151] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-0 >[2018-08-18 10:05:05.690168] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690174] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690188] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690220] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-1 >[2018-08-18 10:05:05.690221] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690279] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690283] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-2 >[2018-08-18 10:05:05.690292] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690288] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690331] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690333] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690340] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690344] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690379] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690393] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-4 >[2018-08-18 10:05:05.690379] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-3 >[2018-08-18 10:05:05.690399] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690389] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690419] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690448] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690471] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690468] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-5 >[2018-08-18 10:05:05.690503] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690514] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690555] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690572] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-6 >[2018-08-18 10:05:05.690624] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690659] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690663] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690687] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690699] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-7 >[2018-08-18 10:05:05.690743] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690778] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690800] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690813] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690821] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-8 >[2018-08-18 10:05:05.690852] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690871] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690876] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690917] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690912] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690958] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690966] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690978] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-9 >[2018-08-18 10:05:05.690988] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.690992] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691003] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-10 >[2018-08-18 10:05:05.691068] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691077] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691096] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691106] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-11 >[2018-08-18 10:05:05.691166] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691191] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691214] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691236] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691260] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691288] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691296] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-13 >[2018-08-18 10:05:05.691295] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691328] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691336] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691352] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691371] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-12 >[2018-08-18 10:05:05.691386] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691405] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691418] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-14 >[2018-08-18 10:05:05.691471] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691497] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691509] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691545] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691559] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-15 >[2018-08-18 10:05:05.691595] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691644] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691651] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691672] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691684] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-16 >[2018-08-18 10:05:05.691730] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691732] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691762] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691790] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691806] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-17 >[2018-08-18 10:05:05.691831] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691843] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691877] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691896] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691905] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-18 >[2018-08-18 10:05:05.691908] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691922] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691967] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691982] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.691994] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-19 >[2018-08-18 10:05:05.692025] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692054] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692070] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692106] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692115] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692109] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692126] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692135] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-20 >[2018-08-18 10:05:05.692163] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692203] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692211] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692225] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-21 >[2018-08-18 10:05:05.692281] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692289] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692311] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692316] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692329] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-22 >[2018-08-18 10:05:05.692373] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692393] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692405] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-23 >[2018-08-18 10:05:05.692416] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692427] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692476] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692494] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692567] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692579] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692592] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692586] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692626] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-25 >[2018-08-18 10:05:05.692629] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692655] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692663] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692682] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-26 >[2018-08-18 10:05:05.692687] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692765] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692790] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692794] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692796] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692838] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692844] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692857] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692856] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692855] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-27 >[2018-08-18 10:05:05.692886] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-28 >[2018-08-18 10:05:05.692909] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692911] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692918] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692953] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692959] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.692971] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-29 >[2018-08-18 10:05:05.692982] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693002] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-30 >[2018-08-18 10:05:05.693023] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693041] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693067] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693085] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693116] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-31 >[2018-08-18 10:05:05.693123] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693122] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693118] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693162] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693159] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693175] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-32 >[2018-08-18 10:05:05.693194] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693201] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693195] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693210] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693224] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-24 >[2018-08-18 10:05:05.693254] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693276] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-33 >[2018-08-18 10:05:05.693305] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693308] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693339] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693361] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693375] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-34 >[2018-08-18 10:05:05.693418] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693419] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693454] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693470] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x352eed0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693478] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-35 >[2018-08-18 10:05:05.693490] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-24 cached_subvol null >[2018-08-18 10:05:05.693506] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001 >[2018-08-18 10:05:05.693513] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x352eed0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693520] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693528] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693536] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.693543] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x352eed0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.694289] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m returned with op_ret 0 >[2018-08-18 10:05:05.696958] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.696979] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m returned with op_ret 0" repeated 35 times between [2018-08-18 10:05:05.694289] and [2018-08-18 10:05:05.696630] >[2018-08-18 10:05:05.696978] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=8c5892ee-79c5-422e-a23b-b0b15f50f3f4, gen=0): picking the request for winding >[2018-08-18 10:05:05.697013] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a at vol0-replicate-29 >[2018-08-18 10:05:05.697218] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=8c5892ee-79c5-422e-a23b-b0b15f50f3f4, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.697465] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a returned with op_ret 0 >[2018-08-18 10:05:05.697626] D [md-cache.c:503:mdc_inode_iatt_set_validate] (-->/usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3cd0) [0x7f3d02157cd0] -->/usr/lib64/glusterfs/3.10.12/xlator/performance/md-cache.so(+0xa802) [0x7f3d01d3e802] -->/usr/lib64/glusterfs/3.10.12/xlator/performance/md-cache.so(+0x66cb) [0x7f3d01d3a6cb] ) 0-vol0-md-cache: discarding the iatt validate request(ctime_nsec) [Invalid argument] >[2018-08-18 10:05:05.697736] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001 on vol0-replicate-24 >[2018-08-18 10:05:05.698095] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.698166] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.698176] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.698217] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.698234] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001 with op_ret -1 [No such file or directory] >[2018-08-18 10:05:05.698250] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001 missing on subvol vol0-replicate-24 >[2018-08-18 10:05:05.698263] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:05.698661] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.698697] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.698722] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.698738] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.698751] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-0 >[2018-08-18 10:05:05.698775] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.698780] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.698808] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.698831] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.698843] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-1 >[2018-08-18 10:05:05.698853] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.698890] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.698888] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.698942] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.698960] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-2 >[2018-08-18 10:05:05.698970] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.698976] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.698998] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699027] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699054] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-3 >[2018-08-18 10:05:05.699055] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699081] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699116] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699158] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699166] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699174] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699174] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-4 >[2018-08-18 10:05:05.699202] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699221] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699230] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-5 >[2018-08-18 10:05:05.699258] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699268] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699271] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699309] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699319] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-6 >[2018-08-18 10:05:05.699362] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699382] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699432] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699456] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699466] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-7 >[2018-08-18 10:05:05.699492] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699516] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699551] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699570] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699575] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699598] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-8 >[2018-08-18 10:05:05.699599] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699655] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699677] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699695] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699705] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-9 >[2018-08-18 10:05:05.699714] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699717] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699741] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699752] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-10 >[2018-08-18 10:05:05.699770] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699800] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699842] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699860] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699869] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-11 >[2018-08-18 10:05:05.699888] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.699927] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700002] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700004] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700006] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700039] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700027] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700063] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-13 >[2018-08-18 10:05:05.700090] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700102] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700111] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-12 >[2018-08-18 10:05:05.700117] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700137] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700162] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700175] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-14 >[2018-08-18 10:05:05.700182] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700221] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700234] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700250] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-11 >[2018-08-18 10:05:05.700263] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700277] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-15 >[2018-08-18 10:05:05.700295] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700301] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 10:05:05.700325] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700370] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700401] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700406] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700428] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700435] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700455] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-16 >[2018-08-18 10:05:05.700475] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700490] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-17 >[2018-08-18 10:05:05.700517] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700554] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700584] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700595] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700603] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-18 >[2018-08-18 10:05:05.700655] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700672] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700678] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700707] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700721] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-19 >[2018-08-18 10:05:05.700738] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700759] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700762] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700775] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700781] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700789] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700801] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700801] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-20 >[2018-08-18 10:05:05.700802] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700820] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700846] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700851] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700864] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700882] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700888] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-22 >[2018-08-18 10:05:05.700885] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-21 >[2018-08-18 10:05:05.700912] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700951] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700966] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.700974] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-23 >[2018-08-18 10:05:05.701044] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701093] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701154] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701176] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701192] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701210] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701220] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-25 >[2018-08-18 10:05:05.701323] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701328] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701331] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701352] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701353] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701387] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-26 >[2018-08-18 10:05:05.701393] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701437] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701469] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701479] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701485] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701488] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-27 >[2018-08-18 10:05:05.701504] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701517] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701525] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-28 >[2018-08-18 10:05:05.701590] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701632] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701637] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701671] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701686] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-29 >[2018-08-18 10:05:05.701739] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701740] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701778] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701799] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701812] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-30 >[2018-08-18 10:05:05.701838] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701838] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701874] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701895] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701899] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701908] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-31 >[2018-08-18 10:05:05.701922] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.701984] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.702009] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.702037] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-32 >[2018-08-18 10:05:05.702065] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.702065] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.702107] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.702140] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.702143] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.702146] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.702186] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-33 >[2018-08-18 10:05:05.702194] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.702187] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.702227] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-24 >[2018-08-18 10:05:05.702232] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.702253] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.702261] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.702286] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.702296] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-35 >[2018-08-18 10:05:05.702572] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.702595] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.702607] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001) from subvol vol0-replicate-34 >[2018-08-18 10:05:05.702622] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-24 cached_subvol null >[2018-08-18 10:05:05.702642] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001 >[2018-08-18 10:05:05.702653] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.702665] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.702679] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.702692] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.702705] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34094cc0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.702831] I [MSGID: 109066] [dht-rename.c:1610:dht_rename] 0-vol0-dht: renaming /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001_2df4b5fea2ce11e89827246e9631871a (hash=vol0-replicate-29/cache=vol0-replicate-29) => /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001 (hash=vol0-replicate-24/cache=<nul>) >[2018-08-18 10:05:05.702952] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=1a59df5f-819d-44a9-8627-99a29744138f, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:05.703079] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=1a59df5f-819d-44a9-8627-99a29744138f, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.703188] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.703600] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-24: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.704112] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.704141] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=1a59df5f-819d-44a9-8627-99a29744138f, gen=0): picking the request for winding >[2018-08-18 10:05:05.704412] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=1a59df5f-819d-44a9-8627-99a29744138f, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.704493] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 10:05:05.704689] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7f3c2c0440d0) allocated with standard calloc() >[2018-08-18 10:05:05.731871] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002 on vol0-replicate-25 >[2018-08-18 10:05:05.732213] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.732265] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.732272] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.732329] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.732348] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002 with op_ret -1 [No such file or directory] >[2018-08-18 10:05:05.732364] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002 missing on subvol vol0-replicate-25 >[2018-08-18 10:05:05.732392] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:05.732796] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.732804] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.732806] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.732859] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.732860] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.732861] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.732859] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.732861] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.732883] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-0 >[2018-08-18 10:05:05.732954] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.732957] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.732964] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.732954] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733001] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733024] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733022] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-1 >[2018-08-18 10:05:05.733022] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733026] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733075] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733076] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-2 >[2018-08-18 10:05:05.733105] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733125] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-3 >[2018-08-18 10:05:05.733127] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733138] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733173] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733183] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733193] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733182] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733223] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-4 >[2018-08-18 10:05:05.733218] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733198] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733265] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733262] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733263] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733288] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-5 >[2018-08-18 10:05:05.733268] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733301] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-6 >[2018-08-18 10:05:05.733303] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733342] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733364] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-7 >[2018-08-18 10:05:05.733365] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733368] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733367] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733428] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733442] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733442] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-8 >[2018-08-18 10:05:05.733459] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733452] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733502] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733506] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733524] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-9 >[2018-08-18 10:05:05.733534] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733524] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733601] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733592] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733630] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733626] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-10 >[2018-08-18 10:05:05.733656] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733680] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733702] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-11 >[2018-08-18 10:05:05.733705] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733710] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733743] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733775] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733781] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733786] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733797] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733820] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-12 >[2018-08-18 10:05:05.733821] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733853] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-13 >[2018-08-18 10:05:05.733858] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733868] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733923] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733923] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733937] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733955] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733973] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733993] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.733985] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-14 >[2018-08-18 10:05:05.734015] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734018] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734053] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-15 >[2018-08-18 10:05:05.734054] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734085] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734098] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-16 >[2018-08-18 10:05:05.734107] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734129] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734137] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734176] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734197] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-17 >[2018-08-18 10:05:05.734213] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734236] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734240] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734262] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734274] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-18 >[2018-08-18 10:05:05.734324] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734364] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734381] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734389] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734418] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734422] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734446] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-19 >[2018-08-18 10:05:05.734427] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734473] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734495] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-20 >[2018-08-18 10:05:05.734501] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734508] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734545] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734550] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734541] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734558] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734593] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734593] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734596] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734616] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-21 >[2018-08-18 10:05:05.734640] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-22 >[2018-08-18 10:05:05.734643] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734658] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734666] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734698] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734705] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734714] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-23 >[2018-08-18 10:05:05.734718] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734740] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734742] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734754] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-24 >[2018-08-18 10:05:05.734764] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734815] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734835] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734842] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734867] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734880] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-26 >[2018-08-18 10:05:05.734973] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.734980] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735058] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735027] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735092] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735100] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735092] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735120] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735119] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735158] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735165] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735165] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735165] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-27 >[2018-08-18 10:05:05.735195] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735195] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-28 >[2018-08-18 10:05:05.735218] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-29 >[2018-08-18 10:05:05.735274] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735269] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735275] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735310] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735278] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735328] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735357] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735352] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735352] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735373] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735375] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-31 >[2018-08-18 10:05:05.735403] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735419] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-32 >[2018-08-18 10:05:05.735408] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-30 >[2018-08-18 10:05:05.735420] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735406] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735462] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735473] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735484] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735494] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735503] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-33 >[2018-08-18 10:05:05.735494] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735523] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735538] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-25 >[2018-08-18 10:05:05.735566] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735593] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735602] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735564] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735633] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735643] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-35 >[2018-08-18 10:05:05.735643] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735666] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-34 >[2018-08-18 10:05:05.735683] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-25 cached_subvol null >[2018-08-18 10:05:05.735695] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002 >[2018-08-18 10:05:05.735708] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735722] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735738] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735752] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.735775] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.736467] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m returned with op_ret 0 >[2018-08-18 10:05:05.738007] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 10:05:05.738125] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-29: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.738510] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:05.738684] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-24: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.739297] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 10:05:05.739409] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-29: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.739803] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:05.739824] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.739995] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.740026] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 10:05:05.740023] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m returned with op_ret 0" repeated 35 times between [2018-08-18 10:05:05.736467] and [2018-08-18 10:05:05.739481] >[2018-08-18 10:05:05.740021] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=1a59df5f-819d-44a9-8627-99a29744138f, gen=0): picking the request for winding >[2018-08-18 10:05:05.740095] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a at vol0-replicate-34 >[2018-08-18 10:05:05.740339] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=1a59df5f-819d-44a9-8627-99a29744138f, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.740440] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a returned with op_ret 0 >[2018-08-18 10:05:05.740522] D [md-cache.c:503:mdc_inode_iatt_set_validate] (-->/usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3cd0) [0x7f3d02157cd0] -->/usr/lib64/glusterfs/3.10.12/xlator/performance/md-cache.so(+0xa802) [0x7f3d01d3e802] -->/usr/lib64/glusterfs/3.10.12/xlator/performance/md-cache.so(+0x66cb) [0x7f3d01d3a6cb] ) 0-vol0-md-cache: discarding the iatt validate request(ctime_nsec) [Invalid argument] >[2018-08-18 10:05:05.740672] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002 on vol0-replicate-25 >[2018-08-18 10:05:05.740902] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.740916] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.740934] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=8c5892ee-79c5-422e-a23b-b0b15f50f3f4, gen=0): picking the request for winding >[2018-08-18 10:05:05.740936] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.740956] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001 at vol0-replicate-29 >[2018-08-18 10:05:05.740941] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.740989] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741005] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002 with op_ret -1 [No such file or directory] >[2018-08-18 10:05:05.741021] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002 missing on subvol vol0-replicate-25 >[2018-08-18 10:05:05.741039] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:05.741159] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=8c5892ee-79c5-422e-a23b-b0b15f50f3f4, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.741277] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001 returned with op_ret 0 >[2018-08-18 10:05:05.741358] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741362] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741403] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741450] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741464] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741466] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-0 >[2018-08-18 10:05:05.741474] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741508] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741524] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741543] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741574] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741583] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741597] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-1 >[2018-08-18 10:05:05.741605] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741603] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-29: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.741618] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-2 >[2018-08-18 10:05:05.741644] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741673] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741687] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741717] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741729] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-3 >[2018-08-18 10:05:05.741751] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741779] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741791] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741813] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741828] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-4 >[2018-08-18 10:05:05.741836] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:05.741886] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741917] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741925] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741966] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.741978] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-5 >[2018-08-18 10:05:05.742018] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742053] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.742036] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742069] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=8c5892ee-79c5-422e-a23b-b0b15f50f3f4, gen=0): picking the request for winding >[2018-08-18 10:05:05.742061] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742087] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742091] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001 at vol0-replicate-29 >[2018-08-18 10:05:05.742096] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-6 >[2018-08-18 10:05:05.742160] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742160] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742235] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742257] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742267] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-7 >[2018-08-18 10:05:05.742303] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=8c5892ee-79c5-422e-a23b-b0b15f50f3f4, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.742309] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742314] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742342] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742346] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742385] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742406] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001 returned with op_ret 0 >[2018-08-18 10:05:05.742416] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742454] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742454] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-8 >[2018-08-18 10:05:05.742456] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742509] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742508] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742537] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-9 >[2018-08-18 10:05:05.742574] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742619] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-29: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.742614] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742618] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742619] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742654] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742667] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-10 >[2018-08-18 10:05:05.742676] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742676] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742698] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-11 >[2018-08-18 10:05:05.742705] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742719] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742741] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742749] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-12 >[2018-08-18 10:05:05.742777] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742782] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742850] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:05.742854] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742873] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742878] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742892] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-13 >[2018-08-18 10:05:05.742921] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742938] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742963] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.742974] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-14 >[2018-08-18 10:05:05.743010] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743050] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743053] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.743062] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743073] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=8c5892ee-79c5-422e-a23b-b0b15f50f3f4, gen=0): picking the request for winding >[2018-08-18 10:05:05.743076] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743095] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-15 >[2018-08-18 10:05:05.743095] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001 at vol0-replicate-29 >[2018-08-18 10:05:05.743104] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743165] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743207] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743244] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743260] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-16 >[2018-08-18 10:05:05.743269] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=8c5892ee-79c5-422e-a23b-b0b15f50f3f4, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.743281] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743284] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743313] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743329] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743335] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743365] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-17 >[2018-08-18 10:05:05.743377] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743390] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743408] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743417] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-18 >[2018-08-18 10:05:05.743431] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743447] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001 returned with op_ret 0 >[2018-08-18 10:05:05.743485] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743489] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743514] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743527] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-19 >[2018-08-18 10:05:05.743543] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743570] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743646] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743677] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743692] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-20 >[2018-08-18 10:05:05.743699] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743728] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-29: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.743741] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743752] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743784] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743788] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743793] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743835] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-21 >[2018-08-18 10:05:05.743837] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743854] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743892] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743902] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743912] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-22 >[2018-08-18 10:05:05.743915] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743939] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743954] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-23 >[2018-08-18 10:05:05.743958] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:05.743968] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.743979] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744019] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744056] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744068] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-24 >[2018-08-18 10:05:05.744116] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744136] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744151] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.744167] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=8c5892ee-79c5-422e-a23b-b0b15f50f3f4, gen=0): picking the request for winding >[2018-08-18 10:05:05.744187] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001 at vol0-replicate-29 >[2018-08-18 10:05:05.744213] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744254] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744294] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744330] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744346] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-26 >[2018-08-18 10:05:05.744359] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=8c5892ee-79c5-422e-a23b-b0b15f50f3f4, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.744356] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744399] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744437] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744468] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744470] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744489] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-27 >[2018-08-18 10:05:05.744504] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001 returned with op_ret 0 >[2018-08-18 10:05:05.744500] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744507] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744535] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744550] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744565] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-28 >[2018-08-18 10:05:05.744561] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744615] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744629] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744627] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744638] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-29 >[2018-08-18 10:05:05.744672] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744669] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744718] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744736] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-30 >[2018-08-18 10:05:05.744737] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744752] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-29: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.744759] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744817] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744840] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744850] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-31 >[2018-08-18 10:05:05.744885] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744900] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744915] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744936] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.744946] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-32 >[2018-08-18 10:05:05.744995] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.745000] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.745008] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:05.745024] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.745060] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.745075] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-33 >[2018-08-18 10:05:05.745105] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.745139] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.745144] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.745148] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.745178] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.745182] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.745202] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-34 >[2018-08-18 10:05:05.745208] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-25 >[2018-08-18 10:05:05.745236] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.745237] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.745257] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.745279] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.745291] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002) from subvol vol0-replicate-35 >[2018-08-18 10:05:05.745303] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-25 cached_subvol null >[2018-08-18 10:05:05.745313] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002 >[2018-08-18 10:05:05.745322] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.745332] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.745349] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.745360] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.745369] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3405bbb0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.745474] I [MSGID: 109066] [dht-rename.c:1610:dht_rename] 0-vol0-dht: renaming /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002_2df7fac0a2ce11e8850b246e9631871a (hash=vol0-replicate-34/cache=vol0-replicate-34) => /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002 (hash=vol0-replicate-25/cache=<nul>) >[2018-08-18 10:05:05.746263] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-25: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.776197] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 10:05:05.776349] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-34: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.776807] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 10:05:05.776919] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-25: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.777409] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 10:05:05.777547] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-34: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.778000] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 10:05:05.778073] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-25: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.778278] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 10:05:05.784918] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.784964] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=1a59df5f-819d-44a9-8627-99a29744138f, gen=0): picking the request for winding >[2018-08-18 10:05:05.785001] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002 at vol0-replicate-34 >[2018-08-18 10:05:05.785330] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=1a59df5f-819d-44a9-8627-99a29744138f, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.785371] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002 returned with op_ret 0 >[2018-08-18 10:05:05.785651] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-34: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.785901] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 10:05:05.786297] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.786328] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=1a59df5f-819d-44a9-8627-99a29744138f, gen=0): picking the request for winding >[2018-08-18 10:05:05.786362] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002 at vol0-replicate-34 >[2018-08-18 10:05:05.786649] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=1a59df5f-819d-44a9-8627-99a29744138f, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.786725] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002 returned with op_ret 0 >[2018-08-18 10:05:05.787064] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-34: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.787335] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 10:05:05.787853] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.787882] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=1a59df5f-819d-44a9-8627-99a29744138f, gen=0): picking the request for winding >[2018-08-18 10:05:05.787915] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002 at vol0-replicate-34 >[2018-08-18 10:05:05.788224] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=1a59df5f-819d-44a9-8627-99a29744138f, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.788271] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002 returned with op_ret 0 >[2018-08-18 10:05:05.788591] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-34: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.788836] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 10:05:05.789212] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.789241] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=1a59df5f-819d-44a9-8627-99a29744138f, gen=0): picking the request for winding >[2018-08-18 10:05:05.789274] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002 at vol0-replicate-34 >[2018-08-18 10:05:05.789570] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=1a59df5f-819d-44a9-8627-99a29744138f, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.789642] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002 returned with op_ret 0 >[2018-08-18 10:05:05.789939] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-34: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.790190] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 10:05:05.802026] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7f3c34057910) as it exceeds the maximum available buffer size >[2018-08-18 10:05:05.803067] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.803094] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=71acb0f5-5f65-4924-9620-75f176411336, gen=0): picking the request for winding >[2018-08-18 10:05:05.803110] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=71acb0f5-5f65-4924-9620-75f176411336, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:05.803384] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.821093] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=71acb0f5-5f65-4924-9620-75f176411336, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:05.821303] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=71acb0f5-5f65-4924-9620-75f176411336, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.821499] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.889696] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 3300593 is serviced using standard calloc() (0x7f3c2c012180) as it exceeds the maximum available buffer size >[2018-08-18 10:05:05.890513] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.890537] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=71acb0f5-5f65-4924-9620-75f176411336, gen=0): picking the request for winding >[2018-08-18 10:05:05.890551] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=71acb0f5-5f65-4924-9620-75f176411336, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:05.890606] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.890620] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 10:05:05.890643] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7f3c34057910) allocated with standard calloc() >[2018-08-18 10:05:05.899170] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.899224] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=1, gfid=71acb0f5-5f65-4924-9620-75f176411336. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=71acb0f5-5f65-4924-9620-75f176411336, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:05.902840] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=71acb0f5-5f65-4924-9620-75f176411336, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:05.903071] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=71acb0f5-5f65-4924-9620-75f176411336, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.903272] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.903299] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=71acb0f5-5f65-4924-9620-75f176411336, gen=1): picking the request for winding >[2018-08-18 10:05:05.903666] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=71acb0f5-5f65-4924-9620-75f176411336, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:05.903785] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 10:05:05.904003] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7f3c2c012180) allocated with standard calloc() >[2018-08-18 10:05:05.905058] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003 on vol0-replicate-14 >[2018-08-18 10:05:05.905461] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.905512] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.905515] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.905579] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.905601] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003 with op_ret -1 [No such file or directory] >[2018-08-18 10:05:05.905617] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003 missing on subvol vol0-replicate-14 >[2018-08-18 10:05:05.905631] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:05.905975] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906021] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906090] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906095] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906115] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906150] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906133] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906172] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-0 >[2018-08-18 10:05:05.906188] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906189] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906214] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-1 >[2018-08-18 10:05:05.906190] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906250] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906283] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906297] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-2 >[2018-08-18 10:05:05.906302] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906335] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906333] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906387] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906412] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-3 >[2018-08-18 10:05:05.906461] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906446] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906446] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906518] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906541] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-4 >[2018-08-18 10:05:05.906588] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906599] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906613] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906611] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906641] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906668] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906700] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-5 >[2018-08-18 10:05:05.906702] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906751] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906746] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906776] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906776] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-6 >[2018-08-18 10:05:05.906814] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906843] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906857] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-7 >[2018-08-18 10:05:05.906905] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906928] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906954] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906966] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.906975] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-8 >[2018-08-18 10:05:05.906997] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907049] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907104] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907121] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907168] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907195] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-9 >[2018-08-18 10:05:05.907205] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907210] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907201] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907212] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907288] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907297] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907304] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907290] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907338] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907361] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-10 >[2018-08-18 10:05:05.907372] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-11 >[2018-08-18 10:05:05.907363] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907362] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907388] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907438] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907450] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907471] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907479] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-12 >[2018-08-18 10:05:05.907498] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907501] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-13 >[2018-08-18 10:05:05.907532] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907657] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907669] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907676] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907731] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907751] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-15 >[2018-08-18 10:05:05.907818] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907831] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907875] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907881] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907914] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907933] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907923] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907921] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907963] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-16 >[2018-08-18 10:05:05.907959] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.907999] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908004] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908023] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-17 >[2018-08-18 10:05:05.908062] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908085] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-18 >[2018-08-18 10:05:05.908087] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908119] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908147] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908165] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908179] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-19 >[2018-08-18 10:05:05.908213] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908218] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908216] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908257] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908279] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-20 >[2018-08-18 10:05:05.908324] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908336] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908354] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908402] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908416] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908429] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-21 >[2018-08-18 10:05:05.908459] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908455] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908495] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908510] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-22 >[2018-08-18 10:05:05.908574] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908574] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908582] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908637] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908651] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908637] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908667] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908682] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-23 >[2018-08-18 10:05:05.908711] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908724] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908734] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-24 >[2018-08-18 10:05:05.908759] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908761] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908804] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908824] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-25 >[2018-08-18 10:05:05.908830] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908860] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908902] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908918] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.908928] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-26 >[2018-08-18 10:05:05.909046] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909049] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909095] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909090] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909089] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909152] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909173] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909164] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909196] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909197] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-28 >[2018-08-18 10:05:05.909234] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909258] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909267] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909270] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-27 >[2018-08-18 10:05:05.909287] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909311] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909309] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909353] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-29 >[2018-08-18 10:05:05.909354] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909390] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-30 >[2018-08-18 10:05:05.909418] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909423] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909423] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909479] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909506] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909508] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909531] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-31 >[2018-08-18 10:05:05.909563] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909592] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909600] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909624] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-32 >[2018-08-18 10:05:05.909660] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909666] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909692] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909704] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-33 >[2018-08-18 10:05:05.909705] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909732] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909755] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909756] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909815] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909815] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909816] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909834] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909839] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-14 >[2018-08-18 10:05:05.909889] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-34 >[2018-08-18 10:05:05.909884] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909928] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.909949] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-35 >[2018-08-18 10:05:05.909975] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-14 cached_subvol null >[2018-08-18 10:05:05.909990] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003 >[2018-08-18 10:05:05.910004] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x341fdc0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.910020] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.910051] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.910068] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.910086] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x341fdc0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.910823] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m returned with op_ret 0 >[2018-08-18 10:05:05.913804] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.913827] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m returned with op_ret 0" repeated 35 times between [2018-08-18 10:05:05.910823] and [2018-08-18 10:05:05.913478] >[2018-08-18 10:05:05.913825] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=71acb0f5-5f65-4924-9620-75f176411336, gen=0): picking the request for winding >[2018-08-18 10:05:05.913868] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a at vol0-replicate-11 >[2018-08-18 10:05:05.914063] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=71acb0f5-5f65-4924-9620-75f176411336, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.914208] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a returned with op_ret 0 >[2018-08-18 10:05:05.914351] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003 on vol0-replicate-14 >[2018-08-18 10:05:05.914587] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.914627] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.914632] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.914688] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.914707] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003 with op_ret -1 [No such file or directory] >[2018-08-18 10:05:05.914722] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003 missing on subvol vol0-replicate-14 >[2018-08-18 10:05:05.914734] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:05.915002] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915048] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915096] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915136] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915153] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915138] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915199] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915204] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-0 >[2018-08-18 10:05:05.915208] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915221] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915223] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915273] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915277] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915298] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-1 >[2018-08-18 10:05:05.915285] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915329] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915330] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-2 >[2018-08-18 10:05:05.915361] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915372] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915404] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915408] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915438] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-3 >[2018-08-18 10:05:05.915442] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915463] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915499] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915507] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915508] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915521] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-4 >[2018-08-18 10:05:05.915563] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915582] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-5 >[2018-08-18 10:05:05.915587] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915614] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915640] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915664] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915676] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-6 >[2018-08-18 10:05:05.915708] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915741] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915768] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915791] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915805] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-7 >[2018-08-18 10:05:05.915828] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915857] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915909] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915947] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915951] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915972] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.915974] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-8 >[2018-08-18 10:05:05.916009] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916042] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916060] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-9 >[2018-08-18 10:05:05.916021] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916066] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916111] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916105] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916140] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916158] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-10 >[2018-08-18 10:05:05.916163] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916178] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916219] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916221] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916249] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916224] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916299] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-11 >[2018-08-18 10:05:05.916308] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916326] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916325] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916326] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916359] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-12 >[2018-08-18 10:05:05.916384] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916406] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-13 >[2018-08-18 10:05:05.916412] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916434] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916511] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916517] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916543] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916577] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916593] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-15 >[2018-08-18 10:05:05.916635] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916671] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916677] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916713] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916717] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916740] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-16 >[2018-08-18 10:05:05.916742] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916749] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916772] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916774] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916784] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916801] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-17 >[2018-08-18 10:05:05.916825] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916848] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916861] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-18 >[2018-08-18 10:05:05.916900] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916921] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916954] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.916999] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917018] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-19 >[2018-08-18 10:05:05.917040] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917018] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917055] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917093] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917106] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-20 >[2018-08-18 10:05:05.917124] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917128] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917153] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917178] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917192] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-21 >[2018-08-18 10:05:05.917206] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917226] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917240] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917259] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917271] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-22 >[2018-08-18 10:05:05.917283] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917322] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917352] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917370] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917394] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917407] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917417] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-23 >[2018-08-18 10:05:05.917445] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917475] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917479] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917480] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917508] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917501] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-24 >[2018-08-18 10:05:05.917538] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917551] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917558] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917573] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-25 >[2018-08-18 10:05:05.917558] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917612] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917645] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-26 >[2018-08-18 10:05:05.917649] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917651] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917653] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917689] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917706] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-27 >[2018-08-18 10:05:05.917721] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917730] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917758] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917782] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917802] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-28 >[2018-08-18 10:05:05.917825] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917868] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917898] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917914] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917952] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917962] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917972] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.917972] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-29 >[2018-08-18 10:05:05.918036] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918038] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918069] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-30 >[2018-08-18 10:05:05.918085] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918085] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918126] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918129] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918140] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-31 >[2018-08-18 10:05:05.918156] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918157] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918202] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918203] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918206] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918232] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-32 >[2018-08-18 10:05:05.918239] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918268] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918284] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-33 >[2018-08-18 10:05:05.918309] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918314] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918318] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918340] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918355] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918363] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-14 >[2018-08-18 10:05:05.918398] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918413] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918435] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-34 >[2018-08-18 10:05:05.918436] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918449] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918483] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918502] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003) from subvol vol0-replicate-35 >[2018-08-18 10:05:05.918520] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-14 cached_subvol null >[2018-08-18 10:05:05.918533] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003 >[2018-08-18 10:05:05.918546] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918561] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918584] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918599] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918613] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c0349b0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:05.918756] I [MSGID: 109066] [dht-rename.c:1610:dht_rename] 0-vol0-dht: renaming /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a (hash=vol0-replicate-11/cache=vol0-replicate-11) => /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003 (hash=vol0-replicate-14/cache=<nul>) >[2018-08-18 10:05:05.919546] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-14: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.941562] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:05.941687] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-11: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.942134] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 10:05:05.942219] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-14: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.942768] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:05.942893] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-11: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:05.943340] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 10:05:05.943387] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.943628] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:05.944378] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.944411] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=71acb0f5-5f65-4924-9620-75f176411336, gen=0): picking the request for winding >[2018-08-18 10:05:05.944447] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003 at vol0-replicate-11 >[2018-08-18 10:05:05.944765] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=71acb0f5-5f65-4924-9620-75f176411336, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.944805] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003 returned with op_ret 0 >[2018-08-18 10:05:05.945145] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.945427] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 10:05:05.945713] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.945740] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=71acb0f5-5f65-4924-9620-75f176411336, gen=0): picking the request for winding >[2018-08-18 10:05:05.945770] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003 at vol0-replicate-11 >[2018-08-18 10:05:05.946065] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=71acb0f5-5f65-4924-9620-75f176411336, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.946104] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003 returned with op_ret 0 >[2018-08-18 10:05:05.946377] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.946615] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 10:05:05.946897] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.946924] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=71acb0f5-5f65-4924-9620-75f176411336, gen=0): picking the request for winding >[2018-08-18 10:05:05.946957] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003 at vol0-replicate-11 >[2018-08-18 10:05:05.947248] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=71acb0f5-5f65-4924-9620-75f176411336, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.947344] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003 returned with op_ret 0 >[2018-08-18 10:05:05.947633] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.947848] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 10:05:05.948155] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:05.948182] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=71acb0f5-5f65-4924-9620-75f176411336, gen=0): picking the request for winding >[2018-08-18 10:05:05.948212] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003 at vol0-replicate-11 >[2018-08-18 10:05:05.948488] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=71acb0f5-5f65-4924-9620-75f176411336, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:05.948563] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003 returned with op_ret 0 >[2018-08-18 10:05:05.948843] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:05.949065] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 10:05:05.971265] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS on vol0-replicate-24 >[2018-08-18 10:05:05.971710] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS with op_ret 0 >[2018-08-18 10:05:05.975991] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt on vol0-replicate-34 >[2018-08-18 10:05:05.976446] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt with op_ret 0 >[2018-08-18 10:05:05.981014] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx on vol0-replicate-20 >[2018-08-18 10:05:05.981404] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx with op_ret 0 >[2018-08-18 10:05:05.984820] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0 on vol0-replicate-23 >[2018-08-18 10:05:05.985247] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0 with op_ret 0 >[2018-08-18 10:05:05.989638] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m on vol0-replicate-22 >[2018-08-18 10:05:05.990037] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m with op_ret 0 >[2018-08-18 10:05:05.994373] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001 on vol0-replicate-24 >[2018-08-18 10:05:05.994770] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001 with op_ret 0 >[2018-08-18 10:05:05.994796] D [MSGID: 0] [dht-common.c:2447:dht_lookup_cbk] 0-vol0-dht: Calling lookup on linkto target vol0-replicate-29 for path /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0001 >[2018-08-18 10:05:05.995158] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-29 >[2018-08-18 10:05:05.995452] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002 on vol0-replicate-25 >[2018-08-18 10:05:05.995812] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002 with op_ret 0 >[2018-08-18 10:05:05.995827] D [MSGID: 0] [dht-common.c:2447:dht_lookup_cbk] 0-vol0-dht: Calling lookup on linkto target vol0-replicate-34 for path /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0002 >[2018-08-18 10:05:05.996180] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-34 >[2018-08-18 10:05:05.996435] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003 on vol0-replicate-14 >[2018-08-18 10:05:05.996839] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003 with op_ret 0 >[2018-08-18 10:05:05.996883] D [MSGID: 0] [dht-common.c:2447:dht_lookup_cbk] 0-vol0-dht: Calling lookup on linkto target vol0-replicate-11 for path /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003 >[2018-08-18 10:05:05.997302] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-11 >[2018-08-18 10:05:06.010746] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171631039154770594.m on vol0-replicate-1 >[2018-08-18 10:05:06.011145] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171631039154770594.m with op_ret 0 >[2018-08-18 10:05:06.015282] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171631039154770594.m returned with op_ret 0 >[2018-08-18 10:05:06.028996] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171631039154770594.m returned with op_ret 0" repeated 107 times between [2018-08-18 10:05:06.015282] and [2018-08-18 10:05:06.026616] >[2018-08-18 10:05:06.028995] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-0: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.029669] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-1: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.030240] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-2: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.030761] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-3: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.031346] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-4: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.031882] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-5: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.032421] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-6: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.032944] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-7: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.033499] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-8: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.034100] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-9: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.034649] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-10: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.035216] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-11: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.035770] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-12: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.036336] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-13: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.036914] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-14: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.037473] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-15: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.038077] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-16: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.038771] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-17: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.039377] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-18: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.039893] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-19: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.040441] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-20: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.040945] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-21: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.041437] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-22: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.041904] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-22 >[2018-08-18 10:05:06.042152] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-23: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.042726] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.043300] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-25: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.043807] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-26: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.044307] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-27: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.044892] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-28: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.045462] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-29: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.045895] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-29 >[2018-08-18 10:05:06.046074] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-30: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.046581] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-31: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.047148] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-32: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.047684] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-33: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.048506] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-34: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.049063] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-35: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.051327] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171631039154770594.m/0002 on vol0-replicate-17 >[2018-08-18 10:05:06.051717] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171631039154770594.m/0002 with op_ret 0 >[2018-08-18 10:05:06.051751] D [MSGID: 0] [dht-common.c:2447:dht_lookup_cbk] 0-vol0-dht: Calling lookup on linkto target vol0-replicate-22 for path /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171631039154770594.m/0002 >[2018-08-18 10:05:06.052170] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-22 >[2018-08-18 10:05:06.052318] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171631039154770594.m/0002 at vol0-replicate-22 >[2018-08-18 10:05:06.052670] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171631039154770594.m/0002 returned with op_ret 0 >[2018-08-18 10:05:06.052969] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-22: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.053999] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-22: lk op is for a transaction >[2018-08-18 10:05:06.054137] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-17: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.054455] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 10:05:06.054602] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171631039154770594.m/0001 on vol0-replicate-16 >[2018-08-18 10:05:06.054991] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171631039154770594.m/0001 with op_ret 0 >[2018-08-18 10:05:06.055024] D [MSGID: 0] [dht-common.c:2447:dht_lookup_cbk] 0-vol0-dht: Calling lookup on linkto target vol0-replicate-29 for path /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171631039154770594.m/0001 >[2018-08-18 10:05:06.055503] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-29 >[2018-08-18 10:05:06.055677] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171631039154770594.m/0001 at vol0-replicate-29 >[2018-08-18 10:05:06.056063] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171631039154770594.m/0001 returned with op_ret 0 >[2018-08-18 10:05:06.056373] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-29: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.058238] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:06.058372] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-16: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.058647] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:06.059127] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171631039154770594.m returned with op_ret 0 >[2018-08-18 10:05:06.064408] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171631039154770594.m returned with op_ret 0" repeated 35 times between [2018-08-18 10:05:06.059127] and [2018-08-18 10:05:06.062181] >[2018-08-18 10:05:06.064407] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-0: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.064526] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-1: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.064588] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-2: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.064656] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-3: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.064719] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-4: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.064797] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-5: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.064855] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-6: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.064915] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-7: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.064976] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-8: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.065044] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-9: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.065111] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-10: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.065187] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-11: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.065244] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-12: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.065285] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-13: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.065341] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-14: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.065392] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-15: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.065430] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-16: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.065488] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-17: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.065529] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-18: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.065587] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-19: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.065633] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-20: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.065711] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-21: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.065753] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-22: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.065795] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-23: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.065832] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.065882] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-25: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.065931] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-26: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.065973] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-27: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.066010] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-28: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.066064] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-29: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.066113] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-30: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.066161] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-31: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.066222] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-32: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.066270] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-33: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.066305] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-34: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.066348] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-35: a08baa35-783b-4980-9ef6-3446ca08fc94: generation now vs cached: 3, 3 >[2018-08-18 10:05:06.074733] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-0: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.074884] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-2: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.074976] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-3: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.075048] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 10:05:06.075125] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-4: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.075169] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 10:05:06.075280] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 10:05:06.075294] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-5: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.075398] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 10:05:06.075417] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-6: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.075614] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 10:05:06.075623] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-7: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.075654] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 10:05:06.075811] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-8: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.075868] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 10:05:06.076122] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-10: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.076123] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 10:05:06.076258] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-11: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.076383] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-12: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.076407] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 10:05:06.076466] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-13: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.076525] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 10:05:06.076616] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-14: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.076652] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 10:05:06.076756] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-15: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.076788] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 10:05:06.076898] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-16: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.076912] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:06.077029] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 10:05:06.077087] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-17: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.077152] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:06.077267] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-18: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.077362] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 10:05:06.077404] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-19: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.077503] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-20: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.077533] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 10:05:06.077669] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-21: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.077714] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 10:05:06.077761] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-20: lk op is for a transaction >[2018-08-18 10:05:06.077834] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-22: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.077950] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:06.078011] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-23: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.078157] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-24: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.078195] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-22: lk op is for a transaction >[2018-08-18 10:05:06.078295] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-25: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.078386] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-23: lk op is for a transaction >[2018-08-18 10:05:06.078415] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 10:05:06.078424] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-26: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.078597] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-27: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.078626] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 10:05:06.078680] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-26: lk op is for a transaction >[2018-08-18 10:05:06.078755] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-28: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.078854] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-27: lk op is for a transaction >[2018-08-18 10:05:06.078918] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-29: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.079048] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 10:05:06.079105] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-30: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.079181] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:06.079222] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-31: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.079411] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-32: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.079422] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-30: lk op is for a transaction >[2018-08-18 10:05:06.079482] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:06.079556] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-33: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.079686] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-34: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.079690] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 10:05:06.079800] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-9: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.079831] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 10:05:06.079854] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-35: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.079910] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 10:05:06.080079] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 10:05:06.080227] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 10:05:06.080342] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-1: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.083331] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 10:05:06.899782] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS on vol0-replicate-24 >[2018-08-18 10:05:06.900318] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS with op_ret 0 >[2018-08-18 10:05:06.904806] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt on vol0-replicate-34 >[2018-08-18 10:05:06.905240] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt with op_ret 0 >[2018-08-18 10:05:06.909737] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx on vol0-replicate-20 >[2018-08-18 10:05:06.910137] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx with op_ret 0 >[2018-08-18 10:05:06.914462] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0 on vol0-replicate-23 >[2018-08-18 10:05:06.914944] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0 with op_ret 0 >[2018-08-18 10:05:06.919472] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m on vol0-replicate-22 >[2018-08-18 10:05:06.919937] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m with op_ret 0 >[2018-08-18 10:05:06.920288] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.920368] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.920414] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.920417] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.920425] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.920467] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.920474] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.920476] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.920524] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.920525] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m on vol0-replicate-1 returned error [No such file or directory] >[2018-08-18 10:05:06.920546] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.920573] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.920595] D [MSGID: 0] [dht-common.c:890:dht_lookup_dir_cbk] 0-vol0-dht: lookup of /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m on vol0-replicate-2 returned error [No such file or directory] >[2018-08-18 10:05:06.920640] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.920717] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.920930] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.920952] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c001f40, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.920992] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c001f40, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.921074] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.921776] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.921826] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c30001f40, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.922364] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c24001f40, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.922383] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-0: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.922410] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2c001f40, vol0-client-0 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.922401] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-2: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.922424] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-1: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.922438] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2c001f40, vol0-client-2 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.922444] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2c001f40, vol0-client-1 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.922640] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.922698] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.922878] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-9: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.922886] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-10: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.922917] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c30001f40, vol0-client-9 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.922899] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-11: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.922929] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c30001f40, vol0-client-10 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.922970] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c30001f40, vol0-client-11 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.923039] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.923172] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.923195] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.923388] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c20001f40, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.923520] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c1c003c30, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.923610] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3000bcb0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.923752] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2400bcb0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.923865] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.924075] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.924199] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.924343] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.924467] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.924598] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.924758] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-21: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.924795] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-36: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.924830] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-38: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.924797] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-22: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.924860] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c3000bcb0, vol0-client-38 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.924844] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c3000bcb0, vol0-client-36 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.924849] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c20001f40, vol0-client-21 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.924901] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-37: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.924916] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c3000bcb0, vol0-client-37 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.924919] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-23: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.924923] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.924937] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c20001f40, vol0-client-22 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.924947] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c20001f40, vol0-client-23 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.924950] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.925062] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c0c001f40, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.925087] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.925116] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c14002dd0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.925431] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-24: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.925458] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c1c003c30, vol0-client-24 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.925470] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-25: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.925510] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c1c003c30, vol0-client-25 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.925588] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c10001f40, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.925634] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.925815] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-40: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.925815] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-39: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.925854] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2400bcb0, vol0-client-40 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.925875] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2400bcb0, vol0-client-39 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.925878] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-41: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.925900] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.925915] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c2400bcb0, vol0-client-41 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.925916] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c20009700, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.926781] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-51: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.926830] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c0c001f40, vol0-client-51 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.926832] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-52: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.926857] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c0c001f40, vol0-client-52 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.926851] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c2c00bcb0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.927088] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-75: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.927143] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c10001f40, vol0-client-75 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.927153] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-76: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.927147] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c30014b80, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.927182] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c10001f40, vol0-client-76 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.927148] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-77: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.927192] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-82: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.927200] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-81: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.927218] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-83: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.927222] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c20009700, vol0-client-82 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.927245] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c20009700, vol0-client-83 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.927251] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c20009700, vol0-client-81 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.927225] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c10001f40, vol0-client-77 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.927843] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-65: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.927837] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-63: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.927855] D [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-vol0-client-64: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.927871] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c14002dd0, vol0-client-65 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.927897] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c14002dd0, vol0-client-64 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.927903] D [MSGID: 0] [client-rpc-fops.c:1650:client3_3_entrylk_cbk] 0-stack-trace: stack-address: 0x7f3c14002dd0, vol0-client-63 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.928818] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c0c00aea0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.928865] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c0c00aea0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.940646] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-vol0-dht: Found anomalies in /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m (gfid = b46b1ae6-8280-42fd-9b29-24a87acdd2c9). Holes=9 overlaps=0 >[2018-08-18 10:05:06.940673] D [MSGID: 0] [dht-common.c:943:dht_lookup_dir_cbk] 0-vol0-dht: fixing assignment on /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m >[2018-08-18 10:05:06.940746] D [MSGID: 0] [dht-selfheal.c:1879:dht_selfheal_layout_new_directory] 0-vol0-dht: chunk size = 0xffffffff / 274638528 = 15.638619 >[2018-08-18 10:05:06.940765] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-12 >[2018-08-18 10:05:06.940775] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-13 >[2018-08-18 10:05:06.940785] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-14 >[2018-08-18 10:05:06.940794] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-15 >[2018-08-18 10:05:06.940803] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-16 >[2018-08-18 10:05:06.940818] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-17 >[2018-08-18 10:05:06.940827] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-18 >[2018-08-18 10:05:06.940836] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-19 >[2018-08-18 10:05:06.940845] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-2 >[2018-08-18 10:05:06.940854] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-20 >[2018-08-18 10:05:06.940863] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-21 >[2018-08-18 10:05:06.940872] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-22 >[2018-08-18 10:05:06.940881] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-23 >[2018-08-18 10:05:06.940890] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-24 >[2018-08-18 10:05:06.940899] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-25 >[2018-08-18 10:05:06.940908] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-26 >[2018-08-18 10:05:06.940917] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-27 >[2018-08-18 10:05:06.940926] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-28 >[2018-08-18 10:05:06.940935] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-29 >[2018-08-18 10:05:06.940944] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-3 >[2018-08-18 10:05:06.940953] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-30 >[2018-08-18 10:05:06.940962] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-31 >[2018-08-18 10:05:06.940971] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-32 >[2018-08-18 10:05:06.940980] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-33 >[2018-08-18 10:05:06.940988] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-34 >[2018-08-18 10:05:06.940998] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-35 >[2018-08-18 10:05:06.941007] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-4 >[2018-08-18 10:05:06.941016] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-5 >[2018-08-18 10:05:06.941025] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-6 >[2018-08-18 10:05:06.941045] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-7 >[2018-08-18 10:05:06.941058] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-8 >[2018-08-18 10:05:06.941067] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-9 >[2018-08-18 10:05:06.941076] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-0 >[2018-08-18 10:05:06.941085] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-1 >[2018-08-18 10:05:06.941094] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-10 >[2018-08-18 10:05:06.941103] D [MSGID: 0] [dht-selfheal.c:1920:dht_selfheal_layout_new_directory] 0-vol0-dht: assigning range size 0x71c71c7 to vol0-replicate-11 >[2018-08-18 10:05:06.951966] D [MSGID: 0] [dht-selfheal.c:1206:dht_selfheal_dir_mkdir_setacl] 0-vol0-dht: ACL_DEFAULT xattr not present >[2018-08-18 10:05:06.951996] D [MSGID: 0] [dht-selfheal.c:1219:dht_selfheal_dir_mkdir_setacl] 0-vol0-dht: ACL_ACCESS xattr not present >[2018-08-18 10:05:06.952010] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m on subvol vol0-replicate-1 >[2018-08-18 10:05:06.952124] D [MSGID: 0] [dht-selfheal.c:1282:dht_selfheal_dir_mkdir_lookup_done] 0-vol0-dht: Creating directory /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m on subvol vol0-replicate-2 >[2018-08-18 10:05:06.952259] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-1: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.952311] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-2: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.952442] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-6: remote operation failed. Path: /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m [File exists] >[2018-08-18 10:05:06.952445] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-3: remote operation failed. Path: /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m [File exists] >[2018-08-18 10:05:06.952475] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-4: remote operation failed. Path: /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m [File exists] >[2018-08-18 10:05:06.952458] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-5: remote operation failed. Path: /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m [File exists] >[2018-08-18 10:05:06.952501] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-8: remote operation failed. Path: /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m [File exists] >[2018-08-18 10:05:06.952498] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c50046ae0, vol0-client-3 returned -1 error: File exists [File exists] >[2018-08-18 10:05:06.952499] D [MSGID: 114031] [client-rpc-fops.c:301:client3_3_mkdir_cbk] 0-vol0-client-7: remote operation failed. Path: /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m [File exists] >[2018-08-18 10:05:06.952471] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c50018170, vol0-client-6 returned -1 error: File exists [File exists] >[2018-08-18 10:05:06.952511] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c50046ae0, vol0-client-4 returned -1 error: File exists [File exists] >[2018-08-18 10:05:06.952569] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c50046ae0, vol0-client-5 returned -1 error: File exists [File exists] >[2018-08-18 10:05:06.952539] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c50018170, vol0-client-8 returned -1 error: File exists [File exists] >[2018-08-18 10:05:06.952556] D [MSGID: 0] [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-stack-trace: stack-address: 0x7f3c50018170, vol0-client-7 returned -1 error: File exists [File exists] >[2018-08-18 10:05:06.952600] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 10:05:06.952621] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 10:05:06.952813] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-1 returned -1 error: File exists [File exists] >[2018-08-18 10:05:06.952827] D [MSGID: 0] [afr-dir-write.c:687:afr_mkdir_unwind] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-2 returned -1 error: File exists [File exists] >[2018-08-18 10:05:06.952838] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m, gfid = b46b1ae6-8280-42fd-9b29-24a87acdd2c9 [File exists] >[2018-08-18 10:05:06.955006] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 10:05:06.952857] D [MSGID: 109005] [dht-selfheal.c:1170:dht_selfheal_dir_mkdir_cbk] 0-vol0-dht: Directory selfheal failed: path = /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m, gfid = b46b1ae6-8280-42fd-9b29-24a87acdd2c9 [File exists] >[2018-08-18 10:05:06.955005] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-0: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.955148] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-10: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.955187] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-11: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.955230] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-12: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.955319] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-0: lk op is for a transaction >[2018-08-18 10:05:06.955324] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.955385] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-10: lk op is for a transaction >[2018-08-18 10:05:06.955389] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.955387] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-1: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.955478] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-11: lk op is for a transaction >[2018-08-18 10:05:06.955487] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-12: lk op is for a transaction >[2018-08-18 10:05:06.955510] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-15: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.955621] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 10:05:06.955629] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.955668] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-17: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.955677] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:06.955700] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-1: lk op is for a transaction >[2018-08-18 10:05:06.955803] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-18: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.955803] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-15: lk op is for a transaction >[2018-08-18 10:05:06.955868] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-19: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.955896] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:06.955928] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-17: lk op is for a transaction >[2018-08-18 10:05:06.956024] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-18: lk op is for a transaction >[2018-08-18 10:05:06.956093] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-19: lk op is for a transaction >[2018-08-18 10:05:06.956134] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-20: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.956195] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.956272] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-22: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.956306] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-23: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.956377] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.956411] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-20: lk op is for a transaction >[2018-08-18 10:05:06.956453] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-25: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.956456] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:06.956541] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-22: lk op is for a transaction >[2018-08-18 10:05:06.956569] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-26: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.956625] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-23: lk op is for a transaction >[2018-08-18 10:05:06.956631] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-27: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.956685] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 10:05:06.956729] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-28: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.956753] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-25: lk op is for a transaction >[2018-08-18 10:05:06.956769] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-29: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.956840] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-26: lk op is for a transaction >[2018-08-18 10:05:06.956877] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-3: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.956881] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-27: lk op is for a transaction >[2018-08-18 10:05:06.956904] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-30: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.956991] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.957006] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-28: lk op is for a transaction >[2018-08-18 10:05:06.957007] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-29: lk op is for a transaction >[2018-08-18 10:05:06.957064] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-32: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.957114] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-30: lk op is for a transaction >[2018-08-18 10:05:06.957141] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-33: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.957163] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-3: lk op is for a transaction >[2018-08-18 10:05:06.957223] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-34: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.957242] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:06.957292] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-32: lk op is for a transaction >[2018-08-18 10:05:06.957307] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-35: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.957384] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.957407] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-33: lk op is for a transaction >[2018-08-18 10:05:06.957441] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-5: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.957447] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-34: lk op is for a transaction >[2018-08-18 10:05:06.957528] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-6: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.957559] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-35: lk op is for a transaction >[2018-08-18 10:05:06.957566] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.957642] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 10:05:06.957679] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-5: lk op is for a transaction >[2018-08-18 10:05:06.957711] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-8: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.957768] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-9: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.957805] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 10:05:06.957812] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-6: lk op is for a transaction >[2018-08-18 10:05:06.957974] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-2: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:06.958007] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-9: lk op is for a transaction >[2018-08-18 10:05:06.958021] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-8: lk op is for a transaction >[2018-08-18 10:05:06.958329] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-2: lk op is for a transaction >[2018-08-18 10:05:06.958457] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-1: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.958467] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-0: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.958465] D [MSGID: 114031] [client-rpc-fops.c:1542:client3_3_inodelk_cbk] 0-vol0-client-2: remote operation failed [Resource temporarily unavailable] >[2018-08-18 10:05:06.958508] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c60c5f320, vol0-client-0 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.958496] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c60c5f320, vol0-client-1 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.958521] D [MSGID: 0] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f3c60c5f320, vol0-client-2 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable] >[2018-08-18 10:05:06.982349] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a on vol0-replicate-4 >[2018-08-18 10:05:06.982668] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.982679] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.982673] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.982735] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.982752] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a with op_ret -1 [No such file or directory] >[2018-08-18 10:05:06.982767] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a missing on subvol vol0-replicate-4 >[2018-08-18 10:05:06.982779] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:06.983071] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983086] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983129] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983138] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983139] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983186] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983152] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983222] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983236] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983244] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983249] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983263] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-0 >[2018-08-18 10:05:06.983278] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983283] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-1 >[2018-08-18 10:05:06.983281] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983317] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-2 >[2018-08-18 10:05:06.983323] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983361] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983384] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983398] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-3 >[2018-08-18 10:05:06.983436] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983497] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983578] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983611] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983627] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983662] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983664] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983680] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983694] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-5 >[2018-08-18 10:05:06.983707] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983726] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983739] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-6 >[2018-08-18 10:05:06.983741] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983773] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983784] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983815] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983828] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-7 >[2018-08-18 10:05:06.983830] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983865] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983910] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983929] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983945] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-8 >[2018-08-18 10:05:06.983960] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.983995] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984007] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984042] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984052] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984063] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-9 >[2018-08-18 10:05:06.984089] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984142] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984164] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984184] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984194] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984207] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-10 >[2018-08-18 10:05:06.984195] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984260] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984279] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-11 >[2018-08-18 10:05:06.984282] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984311] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984346] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984350] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984374] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984392] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984399] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984415] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984425] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-12 >[2018-08-18 10:05:06.984435] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-13 >[2018-08-18 10:05:06.984500] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984532] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984534] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984587] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984597] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984605] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-14 >[2018-08-18 10:05:06.984617] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984609] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984658] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984684] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-15 >[2018-08-18 10:05:06.984714] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984747] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984754] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984813] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984809] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984828] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984812] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984869] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984870] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-16 >[2018-08-18 10:05:06.984876] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984872] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984903] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-17 >[2018-08-18 10:05:06.984898] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984937] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984949] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.984955] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-18 >[2018-08-18 10:05:06.984996] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985013] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985028] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985051] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985074] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-19 >[2018-08-18 10:05:06.985079] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985126] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985165] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985172] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985184] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-20 >[2018-08-18 10:05:06.985181] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985218] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985238] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985251] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-21 >[2018-08-18 10:05:06.985277] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985280] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985320] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985349] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985358] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985379] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-22 >[2018-08-18 10:05:06.985394] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985406] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985433] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985451] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-23 >[2018-08-18 10:05:06.985447] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985497] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985519] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985553] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985567] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-24 >[2018-08-18 10:05:06.985580] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985631] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985633] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985688] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985703] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985704] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-25 >[2018-08-18 10:05:06.985735] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985772] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985795] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985799] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985818] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-26 >[2018-08-18 10:05:06.985831] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985852] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985872] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985880] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985887] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-27 >[2018-08-18 10:05:06.985950] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985958] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.985993] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986002] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986008] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-28 >[2018-08-18 10:05:06.986012] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986071] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986097] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986104] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986118] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986112] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-29 >[2018-08-18 10:05:06.986150] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986169] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986183] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-30 >[2018-08-18 10:05:06.986229] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986225] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986236] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986277] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986299] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-31 >[2018-08-18 10:05:06.986307] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986335] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986337] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986376] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986391] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-32 >[2018-08-18 10:05:06.986407] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986424] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986455] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986455] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986479] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986491] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986488] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986524] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986548] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986554] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-33 >[2018-08-18 10:05:06.986557] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986564] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-34 >[2018-08-18 10:05:06.986550] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986572] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986621] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-4 >[2018-08-18 10:05:06.986625] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986649] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986662] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a) from subvol vol0-replicate-35 >[2018-08-18 10:05:06.986676] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-4 cached_subvol null >[2018-08-18 10:05:06.986689] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a >[2018-08-18 10:05:06.986700] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986711] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986723] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986735] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.986749] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c34027610, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:06.987068] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-0': avail_percent is: 96.00 and avail_space is: 7683143086080 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.987122] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-1': avail_percent is: 95.00 and avail_space is: 7650009186304 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.987168] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-2': avail_percent is: 96.00 and avail_space is: 7682652368896 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.987217] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-3': avail_percent is: 96.00 and avail_space is: 7679969890304 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.987259] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-4': avail_percent is: 96.00 and avail_space is: 7680306479104 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.987311] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-5': avail_percent is: 95.00 and avail_space is: 7674803892224 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.987353] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-6': avail_percent is: 94.00 and avail_space is: 7572583329792 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.987408] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-7': avail_percent is: 95.00 and avail_space is: 7602560430080 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.987458] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-8': avail_percent is: 95.00 and avail_space is: 7674537177088 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.987492] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-9': avail_percent is: 95.00 and avail_space is: 7675679526912 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.987547] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-10': avail_percent is: 94.00 and avail_space is: 7587969724416 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.987580] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-11': avail_percent is: 95.00 and avail_space is: 7666748899328 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.987644] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-12': avail_percent is: 95.00 and avail_space is: 7662860845056 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.987672] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-13': avail_percent is: 95.00 and avail_space is: 7672180158464 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.987702] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-14': avail_percent is: 95.00 and avail_space is: 7671006838784 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.987777] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-15': avail_percent is: 95.00 and avail_space is: 7657363394560 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.987814] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-16': avail_percent is: 96.00 and avail_space is: 7681121935360 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.987840] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-17': avail_percent is: 95.00 and avail_space is: 7670378131456 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.987864] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-18': avail_percent is: 95.00 and avail_space is: 7620654678016 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.987928] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-19': avail_percent is: 95.00 and avail_space is: 7661022814208 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.987981] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-21': avail_percent is: 95.00 and avail_space is: 7663950176256 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.988009] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-20': avail_percent is: 95.00 and avail_space is: 7611642175488 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.988069] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-22': avail_percent is: 95.00 and avail_space is: 7668895387648 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.988118] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-24': avail_percent is: 95.00 and avail_space is: 7665744367616 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.988156] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-23': avail_percent is: 95.00 and avail_space is: 7670234284032 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.988192] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-25': avail_percent is: 95.00 and avail_space is: 7650961227776 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.988232] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-26': avail_percent is: 95.00 and avail_space is: 7659003944960 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.988304] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-27': avail_percent is: 95.00 and avail_space is: 7652757028864 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.988355] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-28': avail_percent is: 95.00 and avail_space is: 7652597972992 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.988390] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-30': avail_percent is: 95.00 and avail_space is: 7678008217600 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.988417] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-29': avail_percent is: 95.00 and avail_space is: 7639883395072 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.988448] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-31': avail_percent is: 95.00 and avail_space is: 7678396715008 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.988480] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-32': avail_percent is: 95.00 and avail_space is: 7664153784320 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.988503] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-33': avail_percent is: 95.00 and avail_space is: 7679374594048 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.988527] D [MSGID: 0] [dht-common.c:6923:dht_create_wind_to_avail_subvol] 0-vol0-dht: creating /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a on vol0-replicate-4 >[2018-08-18 10:05:06.988575] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-34': avail_percent is: 95.00 and avail_space is: 7668147433472 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.988600] D [MSGID: 0] [dht-diskusage.c:96:dht_du_info_cbk] 0-vol0-dht: subvolume 'vol0-replicate-35': avail_percent is: 95.00 and avail_space is: 7666713096192 and avail_inodes is: 99.00 >[2018-08-18 10:05:06.988727] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-4: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:06.989063] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-4 >[2018-08-18 10:05:06.989127] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 10:05:07.349447] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7f3c3401c510) as it exceeds the maximum available buffer size >[2018-08-18 10:05:07.350975] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:07.351012] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=1324fe8f-2b06-4822-aa81-9611da1fa883, gen=0): picking the request for winding >[2018-08-18 10:05:07.351043] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=1324fe8f-2b06-4822-aa81-9611da1fa883, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:07.351387] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:07.368659] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=1324fe8f-2b06-4822-aa81-9611da1fa883, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:07.368875] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=1324fe8f-2b06-4822-aa81-9611da1fa883, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:07.369069] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:07.631857] D [MSGID: 0] [iobuf.c:656:iobuf_get2] 0-iobuf: request for iobuf of size 4194304 is serviced using standard calloc() (0x7f3c3405ae10) as it exceeds the maximum available buffer size >[2018-08-18 10:05:07.635347] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:07.635385] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=1324fe8f-2b06-4822-aa81-9611da1fa883, gen=0): picking the request for winding >[2018-08-18 10:05:07.635405] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=1324fe8f-2b06-4822-aa81-9611da1fa883, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:07.635477] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:07.635495] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 10:05:07.635526] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7f3c3401c510) allocated with standard calloc() >[2018-08-18 10:05:07.647131] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:07.647191] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=1, gfid=1324fe8f-2b06-4822-aa81-9611da1fa883. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=1324fe8f-2b06-4822-aa81-9611da1fa883, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:07.651739] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=1324fe8f-2b06-4822-aa81-9611da1fa883, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:07.651932] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=1324fe8f-2b06-4822-aa81-9611da1fa883, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:07.652120] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:07.652157] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=1324fe8f-2b06-4822-aa81-9611da1fa883, gen=1): picking the request for winding >[2018-08-18 10:05:07.652557] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=1324fe8f-2b06-4822-aa81-9611da1fa883, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:07.652709] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 10:05:07.652939] D [MSGID: 0] [iobuf.c:763:__iobuf_put] 0-iobuf: freeing the iobuf (0x7f3c3405ae10) allocated with standard calloc() >[2018-08-18 10:05:07.654327] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001 on vol0-replicate-21 >[2018-08-18 10:05:07.654649] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.654683] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.654721] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.654745] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.654765] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001 with op_ret -1 [No such file or directory] >[2018-08-18 10:05:07.654777] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001 missing on subvol vol0-replicate-21 >[2018-08-18 10:05:07.654786] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:07.655129] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655139] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655134] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655188] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655198] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655193] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655220] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-0 >[2018-08-18 10:05:07.655242] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655278] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655273] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655296] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655309] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-1 >[2018-08-18 10:05:07.655317] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655327] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655367] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655376] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655398] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-2 >[2018-08-18 10:05:07.655400] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655404] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655443] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655460] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655475] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655499] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655507] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-3 >[2018-08-18 10:05:07.655518] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-4 >[2018-08-18 10:05:07.655526] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655540] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655558] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655591] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655587] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655589] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655621] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655632] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-5 >[2018-08-18 10:05:07.655654] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655656] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655657] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655708] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655712] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-6 >[2018-08-18 10:05:07.655732] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655745] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-7 >[2018-08-18 10:05:07.655775] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655792] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655851] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655873] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655883] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-8 >[2018-08-18 10:05:07.655919] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655922] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655963] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655981] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.655997] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656019] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-9 >[2018-08-18 10:05:07.656027] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656086] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656092] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656127] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656133] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656149] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656161] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-10 >[2018-08-18 10:05:07.656163] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656161] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656198] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656221] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656246] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656248] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656261] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-12 >[2018-08-18 10:05:07.656276] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656277] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656288] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-11 >[2018-08-18 10:05:07.656312] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656338] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-13 >[2018-08-18 10:05:07.656384] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656391] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656385] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656450] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656462] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656491] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-14 >[2018-08-18 10:05:07.656468] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656504] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656560] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656582] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656604] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-15 >[2018-08-18 10:05:07.656616] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656650] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656680] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656686] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656691] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656692] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656733] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-16 >[2018-08-18 10:05:07.656735] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656780] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656781] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-17 >[2018-08-18 10:05:07.656791] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656810] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656838] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656853] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-18 >[2018-08-18 10:05:07.656919] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656940] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656941] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656983] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.656991] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657002] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-19 >[2018-08-18 10:05:07.657038] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657068] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657097] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657100] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657109] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-20 >[2018-08-18 10:05:07.657131] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657267] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657284] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657327] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657361] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657364] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657384] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-22 >[2018-08-18 10:05:07.657422] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657426] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657429] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657468] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657475] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657488] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-23 >[2018-08-18 10:05:07.657480] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657516] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657521] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657521] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657565] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-24 >[2018-08-18 10:05:07.657559] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657569] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657618] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657620] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657642] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-25 >[2018-08-18 10:05:07.657665] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657700] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657721] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-26 >[2018-08-18 10:05:07.657760] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657770] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657800] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657834] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657834] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657834] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657864] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-27 >[2018-08-18 10:05:07.657866] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657904] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657922] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657905] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657920] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657961] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-28 >[2018-08-18 10:05:07.657973] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.657989] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-29 >[2018-08-18 10:05:07.657991] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658048] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658082] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658096] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658107] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-30 >[2018-08-18 10:05:07.658143] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658148] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658184] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658192] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658223] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658254] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-31 >[2018-08-18 10:05:07.658293] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658306] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658312] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658321] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658366] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658382] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658404] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658394] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-32 >[2018-08-18 10:05:07.658392] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658425] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658430] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658412] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658461] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-33 >[2018-08-18 10:05:07.658462] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-21 >[2018-08-18 10:05:07.658491] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658484] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658531] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658535] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658538] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658550] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-34 >[2018-08-18 10:05:07.658592] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658609] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-35 >[2018-08-18 10:05:07.658623] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-21 cached_subvol null >[2018-08-18 10:05:07.658633] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001 >[2018-08-18 10:05:07.658643] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x3411ae0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658654] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658666] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658678] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.658690] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x3411ae0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.659439] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m returned with op_ret 0 >[2018-08-18 10:05:07.663187] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:07.663221] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m returned with op_ret 0" repeated 35 times between [2018-08-18 10:05:07.659439] and [2018-08-18 10:05:07.662715] >[2018-08-18 10:05:07.663220] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=1324fe8f-2b06-4822-aa81-9611da1fa883, gen=0): picking the request for winding >[2018-08-18 10:05:07.663276] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a at vol0-replicate-4 >[2018-08-18 10:05:07.663596] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=1324fe8f-2b06-4822-aa81-9611da1fa883, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:07.663659] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a returned with op_ret 0 >[2018-08-18 10:05:07.663788] D [md-cache.c:493:mdc_inode_iatt_set_validate] (-->/usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3cd0) [0x7f3d02157cd0] -->/usr/lib64/glusterfs/3.10.12/xlator/performance/md-cache.so(+0xa802) [0x7f3d01d3e802] -->/usr/lib64/glusterfs/3.10.12/xlator/performance/md-cache.so(+0x66cb) [0x7f3d01d3a6cb] ) 0-vol0-md-cache: discarding the iatt validate request [Invalid argument] >[2018-08-18 10:05:07.663942] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001 on vol0-replicate-21 >[2018-08-18 10:05:07.664201] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.664243] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.664258] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.664296] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.664313] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001 with op_ret -1 [No such file or directory] >[2018-08-18 10:05:07.664328] D [MSGID: 0] [dht-common.c:2341:dht_lookup_cbk] 0-vol0-dht: Entry /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001 missing on subvol vol0-replicate-21 >[2018-08-18 10:05:07.664340] D [MSGID: 0] [dht-common.c:2112:dht_lookup_everywhere] 0-vol0-dht: winding lookup call to 36 subvols >[2018-08-18 10:05:07.664651] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.664667] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.664670] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.664712] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.664720] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.664754] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-0 >[2018-08-18 10:05:07.664768] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.664774] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.664825] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-1 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.664833] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.664835] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.664844] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-1 >[2018-08-18 10:05:07.664892] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.664920] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-2 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.664934] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-2 >[2018-08-18 10:05:07.664935] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.664981] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.664983] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665042] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-3 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665052] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665082] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-3 >[2018-08-18 10:05:07.665076] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665101] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665128] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665134] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-4 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665146] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665160] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-4 >[2018-08-18 10:05:07.665161] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665202] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-5 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665217] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-5 >[2018-08-18 10:05:07.665254] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665259] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665283] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665327] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-6 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665335] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665349] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-6 >[2018-08-18 10:05:07.665383] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665422] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665448] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-7 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665452] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665459] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-7 >[2018-08-18 10:05:07.665527] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665533] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665566] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665573] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665587] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-8 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665618] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-8 >[2018-08-18 10:05:07.665643] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665670] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-9 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665694] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-9 >[2018-08-18 10:05:07.665713] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665752] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665753] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665799] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665806] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-10 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665806] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665845] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665855] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-10 >[2018-08-18 10:05:07.665881] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-11 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665898] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-11 >[2018-08-18 10:05:07.665922] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-36 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665963] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-37 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665973] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-38 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.665997] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-12 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666011] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-12 >[2018-08-18 10:05:07.666026] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-40 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666064] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-41 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666069] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-39 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666105] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-13 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666120] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-13 >[2018-08-18 10:05:07.666145] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-43 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666158] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-42 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666168] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-44 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666195] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-14 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666207] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-14 >[2018-08-18 10:05:07.666258] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-46 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666260] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-45 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666291] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-47 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666323] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-15 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666350] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-15 >[2018-08-18 10:05:07.666372] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-48 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666381] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-49 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666415] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-50 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666433] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-16 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666444] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-51 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666444] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-16 >[2018-08-18 10:05:07.666486] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-52 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666494] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-53 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666538] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-17 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666547] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-54 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666553] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-17 >[2018-08-18 10:05:07.666594] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-56 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666602] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-55 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666625] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-18 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666630] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-57 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666636] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-18 >[2018-08-18 10:05:07.666660] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-58 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666683] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-59 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666706] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-19 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666719] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-19 >[2018-08-18 10:05:07.666768] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-60 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666796] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-61 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666836] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-62 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666860] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-20 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666870] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-20 >[2018-08-18 10:05:07.666886] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-63 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.666918] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-65 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667026] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-67 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667048] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-68 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667084] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-69 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667086] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-66 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667140] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-70 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667147] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-22 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667161] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-71 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667172] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-72 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667176] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-73 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667171] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-22 >[2018-08-18 10:05:07.667205] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-23 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667217] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-74 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667235] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-23 >[2018-08-18 10:05:07.667250] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-24 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667267] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-24 >[2018-08-18 10:05:07.667276] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-75 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667293] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-76 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667334] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-77 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667335] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-78 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667356] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-79 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667381] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-25 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667399] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-25 >[2018-08-18 10:05:07.667408] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-80 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667437] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-26 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667451] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-26 >[2018-08-18 10:05:07.667478] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-81 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667536] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-82 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667582] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-83 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667632] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-84 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667636] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-27 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667660] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-85 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667671] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-27 >[2018-08-18 10:05:07.667679] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-86 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667718] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-88 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667727] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-87 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667720] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-28 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667755] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-28 >[2018-08-18 10:05:07.667774] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-89 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667807] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-29 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667825] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-91 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667826] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-29 >[2018-08-18 10:05:07.667836] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-90 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667885] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-92 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667903] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-30 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667915] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-30 >[2018-08-18 10:05:07.667917] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-93 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667945] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-94 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.667972] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-95 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668003] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-96 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668010] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-31 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668029] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-31 >[2018-08-18 10:05:07.668081] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-97 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668107] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-98 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668131] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-32 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668146] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-32 >[2018-08-18 10:05:07.668152] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-99 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668174] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-101 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668196] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-100 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668221] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-33 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668229] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-102 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668231] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-33 >[2018-08-18 10:05:07.668268] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-103 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668297] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-104 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668299] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-64 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668340] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-34 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668336] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-105 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668358] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-34 >[2018-08-18 10:05:07.668359] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-21 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668394] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-21 >[2018-08-18 10:05:07.668416] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-106 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668430] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-client-107 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668450] D [MSGID: 0] [afr-common.c:2276:afr_lookup_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-replicate-35 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668461] D [MSGID: 0] [dht-common.c:1914:dht_lookup_everywhere_cbk] 0-vol0-dht: returned with op_ret -1 and op_errno 2 (/FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001) from subvol vol0-replicate-35 >[2018-08-18 10:05:07.668472] D [MSGID: 0] [dht-common.c:1579:dht_lookup_everywhere_done] 0-vol0-dht: STATUS: hashed_subvol vol0-replicate-21 cached_subvol null >[2018-08-18 10:05:07.668480] D [MSGID: 0] [dht-common.c:1640:dht_lookup_everywhere_done] 0-vol0-dht: There was no cached file and unlink on hashed is not skipped /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001 >[2018-08-18 10:05:07.668489] D [MSGID: 0] [dht-common.c:1643:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-dht returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668498] D [MSGID: 0] [write-behind.c:2393:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-write-behind returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668508] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-quick-read returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668524] D [MSGID: 0] [md-cache.c:1048:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0-md-cache returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668534] D [MSGID: 0] [io-stats.c:2186:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f3c3400e7c0, vol0 returned -1 error: No such file or directory [No such file or directory] >[2018-08-18 10:05:07.668670] I [MSGID: 109066] [dht-rename.c:1610:dht_rename] 0-vol0-dht: renaming /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001_2eec2456a2ce11e8ac4d246e9631871a (hash=vol0-replicate-4/cache=vol0-replicate-4) => /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001 (hash=vol0-replicate-21/cache=<nul>) >[2018-08-18 10:05:07.669484] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-21: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:07.670013] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:07.670160] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-4: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:07.670521] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 10:05:07.670624] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-21: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:07.671151] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:07.671269] D [MSGID: 0] [afr-transaction.c:1934:afr_post_nonblocking_entrylk_cbk] 0-vol0-replicate-4: Non blocking entrylks done. Proceeding to FOP >[2018-08-18 10:05:07.671645] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 10:05:07.671675] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-21: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:07.671883] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-21: lk op is for a transaction >[2018-08-18 10:05:07.672562] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:07.672593] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=1324fe8f-2b06-4822-aa81-9611da1fa883, gen=0): picking the request for winding >[2018-08-18 10:05:07.672625] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001 at vol0-replicate-4 >[2018-08-18 10:05:07.672944] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=1324fe8f-2b06-4822-aa81-9611da1fa883, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:07.672973] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001 returned with op_ret 0 >[2018-08-18 10:05:07.673319] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:07.673602] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 10:05:07.673970] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:07.674015] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=1324fe8f-2b06-4822-aa81-9611da1fa883, gen=0): picking the request for winding >[2018-08-18 10:05:07.674055] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001 at vol0-replicate-4 >[2018-08-18 10:05:07.674348] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=1324fe8f-2b06-4822-aa81-9611da1fa883, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:07.674416] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001 returned with op_ret 0 >[2018-08-18 10:05:07.674716] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:07.674917] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 10:05:07.675239] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:07.675267] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=1324fe8f-2b06-4822-aa81-9611da1fa883, gen=0): picking the request for winding >[2018-08-18 10:05:07.675299] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001 at vol0-replicate-4 >[2018-08-18 10:05:07.675586] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=1324fe8f-2b06-4822-aa81-9611da1fa883, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:07.675628] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001 returned with op_ret 0 >[2018-08-18 10:05:07.675867] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:07.676100] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 10:05:07.676395] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:07.676420] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=1324fe8f-2b06-4822-aa81-9611da1fa883, gen=0): picking the request for winding >[2018-08-18 10:05:07.676448] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001 at vol0-replicate-4 >[2018-08-18 10:05:07.676730] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=1324fe8f-2b06-4822-aa81-9611da1fa883, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:07.676770] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /FS/rt/mbXx/service-log_0/c8d9c3a845ed3b27af64ee68dc23fc80_171632311194886716.m/0001 returned with op_ret 0 >[2018-08-18 10:05:07.677012] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-4: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:07.677230] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-4: lk op is for a transaction >[2018-08-18 10:05:11.045882] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.046390] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:11.048183] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7f3d0256d4e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7f3d0d44d631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7f3d02158820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.048245] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 10:05:11.048220] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:11.048244] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-31: ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b: generation now vs cached: 3, 3 >[2018-08-18 10:05:11.048517] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:11.049247] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.049288] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:11.049311] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:11.049587] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:11.050315] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:11.050530] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:11.050834] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.050862] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:11.050889] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:11.050924] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.050968] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:11.050990] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:11.051508] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.051540] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b): ordering.go is not set, hence not winding >[2018-08-18 10:05:11.051560] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 10:05:11.051946] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.052164] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7f3d0256c0b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7f3d0256ca5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 10:05:11.052199] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b): ordering.go is not set, hence not winding >[2018-08-18 10:05:11.052397] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:11.052542] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:11.052690] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.052782] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7f3d0256b6a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7f3d0256ca90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:11.052819] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): picking the request for winding >[2018-08-18 10:05:11.052873] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:11.052886] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:11.053203] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.053328] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:11.053520] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:11.053575] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): picking the request for winding >[2018-08-18 10:05:11.053599] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 10:05:11.053674] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:11.053696] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:11.053703] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.054128] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] )))processing queues >[2018-08-18 10:05:11.054200] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:11.054371] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:11.054415] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): picking the request for winding >[2018-08-18 10:05:11.054436] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 10:05:11.054501] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:11.054520] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:11.054545] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.055071] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.055104] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:11.055263] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): destroying request, removing from all queues >[2018-08-18 10:05:11.055297] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=4): picking the request for winding >[2018-08-18 10:05:11.055398] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.055626] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=4): destroying request, removing from all queues >[2018-08-18 10:05:11.055640] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:11.147607] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.147651] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:11.147740] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /7A/Tp/bSUB/my-bucket_0/79477581a2c811e8bffe246e96318264.t at vol0-replicate-14 >[2018-08-18 10:05:11.148195] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:11.148369] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /7A/Tp/bSUB/my-bucket_0/79477581a2c811e8bffe246e96318264.t returned with op_ret 0 >[2018-08-18 10:05:11.149279] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7f3d0256d4e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7f3d0d44d631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7f3d02158820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.149314] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:11.149348] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-14: f3c71c0d-9173-488b-933b-386804975422: generation now vs cached: 3, 3 >[2018-08-18 10:05:11.149606] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:11.150364] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.150408] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:11.150444] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:11.150765] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:11.151630] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:11.151829] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:11.152090] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.152119] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:11.152160] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:11.152192] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.152275] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:11.152297] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:11.152874] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.152911] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=f3c71c0d-9173-488b-933b-386804975422): ordering.go is not set, hence not winding >[2018-08-18 10:05:11.152934] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 10:05:11.153417] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.153643] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:11.153868] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:11.154172] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7f3d0256c0b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7f3d0256ca5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.154217] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7f3d0256c0b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7f3d0256ca5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 10:05:11.154259] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): picking the request for winding >[2018-08-18 10:05:11.154480] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7f3d0256b6a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7f3d0256ca90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:11.154563] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:11.154586] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:11.155095] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.155191] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:11.155388] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:11.155465] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): picking the request for winding >[2018-08-18 10:05:11.155506] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 10:05:11.155601] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:11.155605] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.155642] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:11.156209] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.156251] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:11.156471] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:11.156514] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): picking the request for winding >[2018-08-18 10:05:11.156537] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 10:05:11.156590] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:11.156607] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:11.156662] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.157242] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.157284] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:11.157517] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): destroying request, removing from all queues >[2018-08-18 10:05:11.157575] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=4): picking the request for winding >[2018-08-18 10:05:11.157728] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:11.157933] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=4): destroying request, removing from all queues >[2018-08-18 10:05:11.158088] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:13.896551] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /vL/jp/KnUC/sjkzhdd_1 on vol0-replicate-24 >[2018-08-18 10:05:13.897114] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /vL/jp/KnUC/sjkzhdd_1 with op_ret 0 >[2018-08-18 10:05:13.902802] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /vL/jp/KnUC/sjkzhdd_1/8d68fb4ea2cb11e88a876c92bf6dccae.t on vol0-replicate-31 >[2018-08-18 10:05:13.903236] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /vL/jp/KnUC/sjkzhdd_1/8d68fb4ea2cb11e88a876c92bf6dccae.t with op_ret 0 >[2018-08-18 10:05:13.903290] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-31 >[2018-08-18 10:05:13.903619] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-31: ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b: generation now vs cached: 3, 3 >[2018-08-18 10:05:13.904670] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:13.904711] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:13.904734] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:13.905069] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:13.905904] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:13.906138] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:13.906341] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:13.906367] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:13.906387] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:13.906445] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:13.906461] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:13.906683] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:13.907373] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:13.907412] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b): ordering.go is not set, hence not winding >[2018-08-18 10:05:13.907434] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 10:05:13.907851] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:13.908066] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7f3d0256c0b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7f3d0256ca5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 10:05:13.908097] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b): ordering.go is not set, hence not winding >[2018-08-18 10:05:13.908296] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7f3d0256b6a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7f3d0256ca90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:13.908584] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:13.908830] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:13.909167] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:13.909201] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): picking the request for winding >[2018-08-18 10:05:13.909221] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): picking the request for winding >[2018-08-18 10:05:13.909235] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 10:05:13.909226] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:13.909313] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:13.909331] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:13.909481] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:13.909642] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:13.909868] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:13.910057] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:13.910146] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b): ordering.go is not set, hence not winding >[2018-08-18 10:05:13.910178] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 10:05:13.910241] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:13.910265] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b): ordering.go is not set, hence not winding >[2018-08-18 10:05:13.910329] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:13.910690] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:13.910727] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:13.910906] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:13.910944] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): picking the request for winding >[2018-08-18 10:05:13.910971] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:13.911029] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:13.911063] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:13.911083] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:13.911114] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:13.911617] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:13.911838] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): destroying request, removing from all queues >[2018-08-18 10:05:13.912054] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:13.912086] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=4): picking the request for winding >[2018-08-18 10:05:13.912456] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=4): destroying request, removing from all queues >[2018-08-18 10:05:13.912617] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:14.111142] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:14.111181] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:14.111230] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /7A/Tp/bSUB/my-bucket_0/79477581a2c811e8bffe246e96318264.t at vol0-replicate-14 >[2018-08-18 10:05:14.111608] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:14.111722] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /7A/Tp/bSUB/my-bucket_0/79477581a2c811e8bffe246e96318264.t returned with op_ret 0 >[2018-08-18 10:05:14.112348] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7f3d0256d4e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7f3d0d44d631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7f3d02158820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:14.112383] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:14.112420] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-14: f3c71c0d-9173-488b-933b-386804975422: generation now vs cached: 3, 3 >[2018-08-18 10:05:14.112675] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:14.113455] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:14.113498] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:14.113524] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:14.113812] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:14.114563] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:14.114765] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:14.114957] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:14.115589] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:14.115627] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:14.115650] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:14.115709] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:14.115728] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:14.116706] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:14.116746] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=f3c71c0d-9173-488b-933b-386804975422): ordering.go is not set, hence not winding >[2018-08-18 10:05:14.116766] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 10:05:14.117211] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:14.117320] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:14.117532] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:14.117732] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:14.117877] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7f3d0256c0b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7f3d0256ca5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 10:05:14.117916] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): picking the request for winding >[2018-08-18 10:05:14.118171] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7f3d0256b6a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7f3d0256ca90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:14.118242] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:14.118263] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:14.118832] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:14.118871] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:14.119084] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:14.119162] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): picking the request for winding >[2018-08-18 10:05:14.119185] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 10:05:14.119241] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:14.119258] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:14.119307] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:14.119704] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:14.119837] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:14.120084] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:14.120165] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): picking the request for winding >[2018-08-18 10:05:14.120199] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 10:05:14.120272] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:14.120291] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:14.120304] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:14.120922] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:14.120964] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:14.121213] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): destroying request, removing from all queues >[2018-08-18 10:05:14.121293] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=4): picking the request for winding >[2018-08-18 10:05:14.121427] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:14.121686] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=4): destroying request, removing from all queues >[2018-08-18 10:05:14.121838] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:15.423913] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:15.423955] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:15.424005] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/sjkzhdd_1/8d68fb4ea2cb11e88a876c92bf6dccae.t at vol0-replicate-31 >[2018-08-18 10:05:15.424397] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:15.424563] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/sjkzhdd_1/8d68fb4ea2cb11e88a876c92bf6dccae.t returned with op_ret 0 >[2018-08-18 10:05:15.425257] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7f3d0256d4e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7f3d0d44d631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7f3d02158820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:15.425293] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:15.425330] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-31: ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b: generation now vs cached: 3, 3 >[2018-08-18 10:05:15.425601] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:15.426452] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:15.426496] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:15.426522] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:15.426895] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:15.427663] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:15.427892] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:15.428118] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:15.431185] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:15.431224] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:15.431249] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:15.431332] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:15.431353] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:15.432871] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:15.432909] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b): ordering.go is not set, hence not winding >[2018-08-18 10:05:15.432929] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 10:05:15.433384] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:15.433597] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7f3d0256c0b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7f3d0256ca5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 10:05:15.433630] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b): ordering.go is not set, hence not winding >[2018-08-18 10:05:15.433817] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7f3d0256b6a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7f3d0256ca90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:15.434185] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:15.434213] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): picking the request for winding >[2018-08-18 10:05:15.434237] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=WRITE, gen=2, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:15.434256] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 10:05:15.434310] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:15.434710] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:15.434742] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=WRITE, gen=2, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:15.434759] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b): ordering.go is not set, hence not winding >[2018-08-18 10:05:15.434773] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 10:05:15.435186] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:15.435419] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:15.435649] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:15.435695] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): picking the request for winding >[2018-08-18 10:05:15.435715] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): picking the request for winding >[2018-08-18 10:05:15.435732] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b. CONFLICT: unique=0, fop=WRITE, gen=1, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:15.435782] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:15.435798] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:15.435875] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:15.435912] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b. CONFLICT: unique=0, fop=WRITE, gen=1, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:15.435927] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:15.436280] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:15.436502] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:15.436699] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:15.436731] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b. CONFLICT: unique=0, fop=WRITE, gen=2, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:15.436803] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:15.437016] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:15.437219] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:15.437391] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:15.437420] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:15.437493] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:15.437700] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:15.437882] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): destroying request, removing from all queues >[2018-08-18 10:05:15.438064] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:15.438091] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=4): picking the request for winding >[2018-08-18 10:05:15.438439] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=4): destroying request, removing from all queues >[2018-08-18 10:05:15.438576] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:16.478417] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:16.478460] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:16.478512] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /7A/Tp/bSUB/my-bucket_0/79477581a2c811e8bffe246e96318264.t at vol0-replicate-14 >[2018-08-18 10:05:16.478855] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:16.478991] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /7A/Tp/bSUB/my-bucket_0/79477581a2c811e8bffe246e96318264.t returned with op_ret 0 >[2018-08-18 10:05:16.479608] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7f3d0256d4e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7f3d0d44d631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7f3d02158820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:16.479640] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:16.479682] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-14: f3c71c0d-9173-488b-933b-386804975422: generation now vs cached: 3, 3 >[2018-08-18 10:05:16.479915] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:16.480658] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:16.480702] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:16.480740] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:16.481063] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:16.481973] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:16.482186] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:16.482375] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:16.484814] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:16.484845] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:16.484864] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:16.484913] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:16.484930] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:16.486289] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:16.486319] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=f3c71c0d-9173-488b-933b-386804975422): ordering.go is not set, hence not winding >[2018-08-18 10:05:16.486336] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 10:05:16.486703] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:16.486903] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7f3d0256c0b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7f3d0256ca5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 10:05:16.486930] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=f3c71c0d-9173-488b-933b-386804975422): ordering.go is not set, hence not winding >[2018-08-18 10:05:16.487115] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7f3d0256b6a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7f3d0256ca90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:16.487456] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:16.487480] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): picking the request for winding >[2018-08-18 10:05:16.487504] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=WRITE, gen=2, gfid=f3c71c0d-9173-488b-933b-386804975422. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=f3c71c0d-9173-488b-933b-386804975422, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:16.487517] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 10:05:16.487567] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:16.487934] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:16.487964] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=WRITE, gen=2, gfid=f3c71c0d-9173-488b-933b-386804975422. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=f3c71c0d-9173-488b-933b-386804975422, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:16.487984] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=f3c71c0d-9173-488b-933b-386804975422): ordering.go is not set, hence not winding >[2018-08-18 10:05:16.487997] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 10:05:16.488381] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:16.488483] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:16.488662] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:16.488734] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): picking the request for winding >[2018-08-18 10:05:16.488766] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): picking the request for winding >[2018-08-18 10:05:16.488785] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=f3c71c0d-9173-488b-933b-386804975422. CONFLICT: unique=0, fop=WRITE, gen=1, gfid=f3c71c0d-9173-488b-933b-386804975422, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:16.488838] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:16.488882] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 10:05:16.488867] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=f3c71c0d-9173-488b-933b-386804975422. CONFLICT: unique=0, fop=WRITE, gen=1, gfid=f3c71c0d-9173-488b-933b-386804975422, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:16.488881] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:16.488925] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:16.489134] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:16.489304] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:16.489416] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 10:05:16.489445] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:16.488956] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:16.489472] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=f3c71c0d-9173-488b-933b-386804975422. CONFLICT: unique=0, fop=WRITE, gen=2, gfid=f3c71c0d-9173-488b-933b-386804975422, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:16.489415] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:16.489682] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:16.489824] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:16.489967] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:16.489989] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=f3c71c0d-9173-488b-933b-386804975422. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=f3c71c0d-9173-488b-933b-386804975422, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:16.490054] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:16.490220] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:16.490358] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): destroying request, removing from all queues >[2018-08-18 10:05:16.490492] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:16.490510] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=4): picking the request for winding >[2018-08-18 10:05:16.490767] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=4): destroying request, removing from all queues >[2018-08-18 10:05:16.490869] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:19.097754] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /vL/jp/KnUC/cpsbxt_5 on vol0-replicate-3 >[2018-08-18 10:05:19.098304] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /vL/jp/KnUC/cpsbxt_5 with op_ret 0 >[2018-08-18 10:05:19.103143] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /vL/jp/KnUC/cpsbxt_5/5113626ba2cd11e897a0246e96313454.t on vol0-replicate-13 >[2018-08-18 10:05:19.103593] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /vL/jp/KnUC/cpsbxt_5/5113626ba2cd11e897a0246e96313454.t with op_ret 0 >[2018-08-18 10:05:19.103629] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-13 >[2018-08-18 10:05:19.104173] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-13: 80d1ddf0-c466-4ba8-969e-22f737923bb5: generation now vs cached: 3, 3 >[2018-08-18 10:05:19.105292] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:19.105331] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=0): picking the request for winding >[2018-08-18 10:05:19.105364] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:19.105626] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:19.106427] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:19.106635] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:19.106828] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:19.107430] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:19.107469] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=0): picking the request for winding >[2018-08-18 10:05:19.107491] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:19.107549] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:19.107568] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 10:05:19.108622] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:19.108657] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5): ordering.go is not set, hence not winding >[2018-08-18 10:05:19.108676] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 10:05:19.109131] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:19.109875] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7f3d0256c0b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7f3d0256ca5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 10:05:19.109907] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5): ordering.go is not set, hence not winding >[2018-08-18 10:05:19.110216] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:19.110447] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:19.110645] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:19.110788] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7f3d0256b6a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7f3d0256ca90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:19.110842] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=1): picking the request for winding >[2018-08-18 10:05:19.110922] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:19.110940] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 10:05:19.111265] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:19.111488] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:19.111701] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:19.111778] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=2): picking the request for winding >[2018-08-18 10:05:19.111817] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 10:05:19.111883] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:19.111902] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 10:05:19.111898] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:19.112453] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:19.112492] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:19.112688] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:19.112728] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=3): picking the request for winding >[2018-08-18 10:05:19.112756] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 10:05:19.112816] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-13: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:19.112833] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 10:05:19.112868] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:19.113478] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:19.113525] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:19.113727] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=3): destroying request, removing from all queues >[2018-08-18 10:05:19.113768] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=4): picking the request for winding >[2018-08-18 10:05:19.113924] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:19.114145] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=80d1ddf0-c466-4ba8-969e-22f737923bb5, gen=4): destroying request, removing from all queues >[2018-08-18 10:05:19.114291] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-13: lk op is for a transaction >[2018-08-18 10:05:22.297072] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-3: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297126] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-4: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297139] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-0: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297151] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-1: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297162] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-2: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297174] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-6: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297185] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-7: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297196] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-8: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297206] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-9: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297216] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-10: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297225] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-11: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297235] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-12: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297246] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-13: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297258] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-14: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297269] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-15: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297279] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-16: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297290] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-17: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297301] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-18: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297314] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-19: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297326] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-20: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297337] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-21: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297355] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-22: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297365] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-23: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297375] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-24: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297386] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-25: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297397] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-26: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297418] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-27: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297427] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-28: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297435] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-29: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297443] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-30: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297453] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-31: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297463] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-32: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297472] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-33: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297482] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-34: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297492] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-35: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297502] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-36: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297514] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-37: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297525] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-38: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297534] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-39: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297542] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-40: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297551] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-41: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297559] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-42: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297568] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-43: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297578] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-44: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297588] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-45: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297601] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-46: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297610] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-47: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297619] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-48: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297627] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-49: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297638] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-50: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297649] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-51: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297659] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-52: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297669] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-53: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297680] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-54: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297690] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-55: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297701] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-56: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297711] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-57: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297721] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-58: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297729] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-59: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297738] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-60: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297752] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-61: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297762] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-62: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297772] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-63: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297781] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-64: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297790] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-65: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297798] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-66: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297807] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-67: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297816] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-68: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297828] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-69: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297839] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-70: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297853] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-71: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297863] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-72: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297873] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-73: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297882] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-74: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297893] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-75: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297903] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-76: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297911] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-77: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297920] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-78: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297929] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-79: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297938] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-80: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297947] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-81: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297957] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-82: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297966] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-83: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297974] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-84: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297983] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-85: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.297994] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-86: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.298005] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-87: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.298016] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-88: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.298027] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-89: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.298052] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-90: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.298062] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-91: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.298071] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-92: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.298081] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-93: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.298091] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-94: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.298105] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-95: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.298113] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-96: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.298122] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-97: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.298131] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-98: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.298140] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-99: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.298151] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-100: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.298162] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-101: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.298170] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-102: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.298181] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-103: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.298191] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-104: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.298202] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-105: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.298213] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-106: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.298221] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-5: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:22.298230] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-107: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.833867] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-9: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.833934] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-10: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.833952] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-0: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.833982] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-1: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.833996] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-2: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834010] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-3: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834044] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-4: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834068] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-5: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834093] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-6: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834114] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-7: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834140] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-8: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834159] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-12: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834178] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-13: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834199] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-14: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834226] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-15: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834240] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-16: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834256] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-17: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834272] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-18: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834290] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-19: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834307] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-20: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834342] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-21: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834364] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-22: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834386] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-23: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834408] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-24: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834430] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-25: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834452] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-26: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834473] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-27: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834498] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-28: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834520] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-29: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834548] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-30: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834570] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-31: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834600] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-32: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834615] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-33: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834633] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-34: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834655] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-35: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834673] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-36: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834690] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-37: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834706] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-38: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834726] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-39: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834751] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-40: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834771] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-41: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834792] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-42: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834814] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-43: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834839] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-44: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834870] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-45: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834891] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-46: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834911] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-47: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834932] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-48: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834952] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-49: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834968] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-50: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.834985] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-51: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835001] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-52: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835015] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-53: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835038] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-54: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835057] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-55: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835079] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-56: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835101] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-57: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835126] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-58: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835150] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-59: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835172] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-60: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835194] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-61: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835215] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-62: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835236] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-66: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835271] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-67: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835286] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-68: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835302] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-69: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835317] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-70: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835342] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-71: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835358] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-72: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835374] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-73: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835396] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-74: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835416] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-75: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835437] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-76: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835463] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-77: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835489] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-78: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835517] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-79: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835544] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-80: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835570] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-81: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835592] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-82: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835620] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-83: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835643] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-84: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835659] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-85: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835680] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-86: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835696] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-87: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835717] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-88: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835731] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-89: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835747] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-90: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835770] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-91: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835796] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-92: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835816] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-93: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835837] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-94: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835857] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-95: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835882] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-96: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835902] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-97: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835929] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-98: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835949] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-99: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835970] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-100: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.835990] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-101: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.836004] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-102: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.836020] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-103: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.836043] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-104: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.836060] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-105: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.836078] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-106: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.836098] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-11: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:27.836114] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-107: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:32.635703] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /vL on vol0-replicate-17 >[2018-08-18 10:05:32.636206] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49205: ping timer event already removed >[2018-08-18 10:05:32.636590] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49205: ping timer event already removed >[2018-08-18 10:05:32.636889] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49205: ping timer event already removed >[2018-08-18 10:05:32.637029] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /vL with op_ret 0 >[2018-08-18 10:05:32.637469] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49188: ping timer event already removed >[2018-08-18 10:05:32.637743] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49188: ping timer event already removed >[2018-08-18 10:05:32.637989] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49188: ping timer event already removed >[2018-08-18 10:05:32.638338] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49189: ping timer event already removed >[2018-08-18 10:05:32.638615] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49189: ping timer event already removed >[2018-08-18 10:05:32.638860] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49189: ping timer event already removed >[2018-08-18 10:05:32.640026] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49190: ping timer event already removed >[2018-08-18 10:05:32.640467] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49190: ping timer event already removed >[2018-08-18 10:05:32.640739] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49190: ping timer event already removed >[2018-08-18 10:05:32.641085] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49191: ping timer event already removed >[2018-08-18 10:05:32.641341] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49191: ping timer event already removed >[2018-08-18 10:05:32.641576] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49191: ping timer event already removed >[2018-08-18 10:05:32.641870] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49192: ping timer event already removed >[2018-08-18 10:05:32.642139] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49192: ping timer event already removed >[2018-08-18 10:05:32.642416] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49192: ping timer event already removed >[2018-08-18 10:05:32.642689] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49193: ping timer event already removed >[2018-08-18 10:05:32.642917] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49193: ping timer event already removed >[2018-08-18 10:05:32.643165] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49193: ping timer event already removed >[2018-08-18 10:05:32.643471] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49194: ping timer event already removed >[2018-08-18 10:05:32.643703] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49194: ping timer event already removed >[2018-08-18 10:05:32.643927] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49194: ping timer event already removed >[2018-08-18 10:05:32.644215] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49195: ping timer event already removed >[2018-08-18 10:05:32.644466] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49195: ping timer event already removed >[2018-08-18 10:05:32.644725] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49195: ping timer event already removed >[2018-08-18 10:05:32.644990] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49196: ping timer event already removed >[2018-08-18 10:05:32.645252] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49196: ping timer event already removed >[2018-08-18 10:05:32.645469] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49196: ping timer event already removed >[2018-08-18 10:05:32.645725] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49197: ping timer event already removed >[2018-08-18 10:05:32.645943] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49197: ping timer event already removed >[2018-08-18 10:05:32.646169] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49197: ping timer event already removed >[2018-08-18 10:05:32.646457] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49198: ping timer event already removed >[2018-08-18 10:05:32.646674] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49198: ping timer event already removed >[2018-08-18 10:05:32.646919] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49198: ping timer event already removed >[2018-08-18 10:05:32.647193] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49199: ping timer event already removed >[2018-08-18 10:05:32.647417] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49199: ping timer event already removed >[2018-08-18 10:05:32.647629] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49199: ping timer event already removed >[2018-08-18 10:05:32.647900] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49200: ping timer event already removed >[2018-08-18 10:05:32.648123] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49200: ping timer event already removed >[2018-08-18 10:05:32.648349] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49200: ping timer event already removed >[2018-08-18 10:05:32.648595] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49201: ping timer event already removed >[2018-08-18 10:05:32.648809] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49201: ping timer event already removed >[2018-08-18 10:05:32.649039] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49201: ping timer event already removed >[2018-08-18 10:05:32.649305] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49202: ping timer event already removed >[2018-08-18 10:05:32.649601] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49202: ping timer event already removed >[2018-08-18 10:05:32.649831] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49202: ping timer event already removed >[2018-08-18 10:05:32.650148] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49203: ping timer event already removed >[2018-08-18 10:05:32.650418] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49203: ping timer event already removed >[2018-08-18 10:05:32.650652] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49203: ping timer event already removed >[2018-08-18 10:05:32.651108] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49204: ping timer event already removed >[2018-08-18 10:05:32.651419] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49204: ping timer event already removed >[2018-08-18 10:05:32.651653] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49204: ping timer event already removed >[2018-08-18 10:05:32.652054] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49206: ping timer event already removed >[2018-08-18 10:05:32.652301] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49206: ping timer event already removed >[2018-08-18 10:05:32.652519] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49206: ping timer event already removed >[2018-08-18 10:05:32.652783] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49207: ping timer event already removed >[2018-08-18 10:05:32.653005] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49207: ping timer event already removed >[2018-08-18 10:05:32.653292] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49207: ping timer event already removed >[2018-08-18 10:05:32.653566] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49208: ping timer event already removed >[2018-08-18 10:05:32.653807] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49208: ping timer event already removed >[2018-08-18 10:05:32.654021] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49208: ping timer event already removed >[2018-08-18 10:05:32.654329] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49209: ping timer event already removed >[2018-08-18 10:05:32.654537] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49209: ping timer event already removed >[2018-08-18 10:05:32.654744] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49209: ping timer event already removed >[2018-08-18 10:05:32.654981] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49210: ping timer event already removed >[2018-08-18 10:05:32.655186] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49210: ping timer event already removed >[2018-08-18 10:05:32.655401] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49210: ping timer event already removed >[2018-08-18 10:05:32.655664] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49211: ping timer event already removed >[2018-08-18 10:05:32.655926] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49211: ping timer event already removed >[2018-08-18 10:05:32.656134] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49211: ping timer event already removed >[2018-08-18 10:05:32.656373] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49212: ping timer event already removed >[2018-08-18 10:05:32.656581] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49212: ping timer event already removed >[2018-08-18 10:05:32.656776] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49212: ping timer event already removed >[2018-08-18 10:05:32.657019] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49213: ping timer event already removed >[2018-08-18 10:05:32.657246] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49213: ping timer event already removed >[2018-08-18 10:05:32.657445] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49213: ping timer event already removed >[2018-08-18 10:05:32.657678] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49214: ping timer event already removed >[2018-08-18 10:05:32.657882] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49214: ping timer event already removed >[2018-08-18 10:05:32.658099] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49214: ping timer event already removed >[2018-08-18 10:05:32.658367] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49215: ping timer event already removed >[2018-08-18 10:05:32.658566] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49215: ping timer event already removed >[2018-08-18 10:05:32.658812] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49215: ping timer event already removed >[2018-08-18 10:05:32.659057] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49216: ping timer event already removed >[2018-08-18 10:05:32.659256] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49216: ping timer event already removed >[2018-08-18 10:05:32.659462] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49216: ping timer event already removed >[2018-08-18 10:05:32.659693] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49217: ping timer event already removed >[2018-08-18 10:05:32.659900] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49217: ping timer event already removed >[2018-08-18 10:05:32.660107] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49217: ping timer event already removed >[2018-08-18 10:05:32.660345] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49218: ping timer event already removed >[2018-08-18 10:05:32.660548] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49218: ping timer event already removed >[2018-08-18 10:05:32.660745] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49218: ping timer event already removed >[2018-08-18 10:05:32.660981] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49219: ping timer event already removed >[2018-08-18 10:05:32.661184] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49219: ping timer event already removed >[2018-08-18 10:05:32.661386] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49219: ping timer event already removed >[2018-08-18 10:05:32.661632] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49220: ping timer event already removed >[2018-08-18 10:05:32.661829] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49220: ping timer event already removed >[2018-08-18 10:05:32.662025] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49220: ping timer event already removed >[2018-08-18 10:05:32.662275] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49221: ping timer event already removed >[2018-08-18 10:05:32.662510] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49221: ping timer event already removed >[2018-08-18 10:05:32.662718] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49221: ping timer event already removed >[2018-08-18 10:05:32.662969] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49222: ping timer event already removed >[2018-08-18 10:05:32.663182] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49222: ping timer event already removed >[2018-08-18 10:05:32.663386] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49222: ping timer event already removed >[2018-08-18 10:05:32.663630] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49223: ping timer event already removed >[2018-08-18 10:05:32.663834] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49223: ping timer event already removed >[2018-08-18 10:05:32.664045] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49223: ping timer event already removed >[2018-08-18 10:05:32.664476] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /vL/jp on vol0-replicate-19 >[2018-08-18 10:05:32.664884] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /vL/jp with op_ret 0 >[2018-08-18 10:05:32.669382] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /vL/jp/KnUC on vol0-replicate-13 >[2018-08-18 10:05:32.669794] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /vL/jp/KnUC with op_ret 0 >[2018-08-18 10:05:32.674342] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /vL/jp/KnUC/sjkzhdd_1 on vol0-replicate-24 >[2018-08-18 10:05:32.674764] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /vL/jp/KnUC/sjkzhdd_1 with op_ret 0 >[2018-08-18 10:05:32.678949] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /vL/jp/KnUC/sjkzhdd_1/8d68fb4ea2cb11e88a876c92bf6dccae.t on vol0-replicate-31 >[2018-08-18 10:05:32.679341] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /vL/jp/KnUC/sjkzhdd_1/8d68fb4ea2cb11e88a876c92bf6dccae.t with op_ret 0 >[2018-08-18 10:05:32.679385] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-31 >[2018-08-18 10:05:32.679995] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-31: ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b: generation now vs cached: 3, 3 >[2018-08-18 10:05:32.681248] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.681290] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:32.681313] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:32.681610] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:32.682310] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.682460] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:32.682699] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:32.682780] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): picking the request for winding >[2018-08-18 10:05:32.682815] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 10:05:32.682898] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:32.682951] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:32.682973] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.683698] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:32.683900] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:32.684145] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.684176] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:32.684197] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:32.684298] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:32.684318] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:32.684328] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.684381] D [socket.c:532:__socket_rwv] 0-vol0-client-93: would have passed zero length to read/write >[2018-08-18 10:05:32.684414] D [socket.c:532:__socket_rwv] 0-vol0-client-94: would have passed zero length to read/write >[2018-08-18 10:05:32.684437] D [socket.c:532:__socket_rwv] 0-vol0-client-95: would have passed zero length to read/write >[2018-08-18 10:05:32.684758] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] )))processing queues >[2018-08-18 10:05:32.684832] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:32.685001] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:32.685045] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): picking the request for winding >[2018-08-18 10:05:32.685070] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 10:05:32.685119] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:32.685135] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:32.685182] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.685690] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.685755] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:32.685991] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:32.686045] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): picking the request for winding >[2018-08-18 10:05:32.686068] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 10:05:32.686153] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:32.686175] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:32.686233] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.686612] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.686728] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:32.686958] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:32.687004] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): picking the request for winding >[2018-08-18 10:05:32.687025] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 10:05:32.687093] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:32.687110] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:32.687196] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.687604] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.687701] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:32.687896] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): destroying request, removing from all queues >[2018-08-18 10:05:32.687938] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=4): picking the request for winding >[2018-08-18 10:05:32.688094] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.688281] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=4): destroying request, removing from all queues >[2018-08-18 10:05:32.688422] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:32.822257] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /7A on vol0-replicate-1 >[2018-08-18 10:05:32.822757] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /7A with op_ret 0 >[2018-08-18 10:05:32.831028] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /7A/Tp on vol0-replicate-35 >[2018-08-18 10:05:32.831437] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /7A/Tp with op_ret 0 >[2018-08-18 10:05:32.836251] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /7A/Tp/bSUB on vol0-replicate-18 >[2018-08-18 10:05:32.836637] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /7A/Tp/bSUB with op_ret 0 >[2018-08-18 10:05:32.840903] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /7A/Tp/bSUB/my-bucket_0 on vol0-replicate-11 >[2018-08-18 10:05:32.841293] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /7A/Tp/bSUB/my-bucket_0 with op_ret 0 >[2018-08-18 10:05:32.845734] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /7A/Tp/bSUB/my-bucket_0/79477581a2c811e8bffe246e96318264.t on vol0-replicate-14 >[2018-08-18 10:05:32.846123] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /7A/Tp/bSUB/my-bucket_0/79477581a2c811e8bffe246e96318264.t with op_ret 0 >[2018-08-18 10:05:32.846159] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-14 >[2018-08-18 10:05:32.846880] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-14: f3c71c0d-9173-488b-933b-386804975422: generation now vs cached: 3, 3 >[2018-08-18 10:05:32.848138] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.848174] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:32.848194] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:32.848492] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:32.849144] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.849275] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:32.849480] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:32.849523] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): picking the request for winding >[2018-08-18 10:05:32.849551] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 10:05:32.849611] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:32.849627] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:32.849685] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.850434] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:32.850663] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:32.851025] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.851049] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.851083] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:32.851105] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:32.851185] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:32.851212] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:32.851270] D [socket.c:532:__socket_rwv] 0-vol0-client-42: would have passed zero length to read/write >[2018-08-18 10:05:32.851302] D [socket.c:532:__socket_rwv] 0-vol0-client-43: would have passed zero length to read/write >[2018-08-18 10:05:32.851326] D [socket.c:532:__socket_rwv] 0-vol0-client-44: would have passed zero length to read/write >[2018-08-18 10:05:32.851522] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.851754] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:32.851982] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:32.852024] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): picking the request for winding >[2018-08-18 10:05:32.852056] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 10:05:32.852106] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:32.852121] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:32.852188] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.852668] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.852682] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:32.852886] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:32.852923] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): picking the request for winding >[2018-08-18 10:05:32.852942] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 10:05:32.853015] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:32.853038] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:32.853082] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.853444] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.853577] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:32.853789] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:32.853837] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): picking the request for winding >[2018-08-18 10:05:32.853856] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 10:05:32.853903] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:32.853919] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:32.853986] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.854404] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.854519] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:32.854743] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): destroying request, removing from all queues >[2018-08-18 10:05:32.854785] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=4): picking the request for winding >[2018-08-18 10:05:32.854937] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:32.855152] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=4): destroying request, removing from all queues >[2018-08-18 10:05:32.855328] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:35.768017] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-51: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768092] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-52: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768114] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-0: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768154] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-1: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768164] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-2: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768181] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-3: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768191] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-4: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768200] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-5: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768209] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-6: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768221] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-7: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768232] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-8: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768241] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-9: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768251] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-10: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768270] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-11: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768281] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-12: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768292] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-13: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768303] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-14: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768321] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-15: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768339] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-16: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768356] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-17: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768372] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-18: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768389] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-19: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768413] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-20: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768429] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-21: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768438] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-22: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768447] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-23: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768456] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-24: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768468] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-25: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768479] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-26: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768488] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-27: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768497] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-28: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768506] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-29: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768515] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-30: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768525] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-31: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768536] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-32: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768545] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-33: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768560] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-34: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768576] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-35: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768592] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-36: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768609] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-37: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768633] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-38: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768642] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-39: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768656] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-40: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768665] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-41: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768674] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-42: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768683] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-43: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768693] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-44: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768702] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-45: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768711] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-46: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768720] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-47: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768729] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-48: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768739] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-49: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768749] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-50: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768771] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-54: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768787] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-55: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768804] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-56: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768814] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-57: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768823] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-58: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768832] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-59: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768841] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-60: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768850] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-61: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768860] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-62: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768871] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-63: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768880] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-64: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768889] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-65: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768898] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-66: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768911] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-67: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768920] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-68: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768937] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-69: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768954] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-70: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768969] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-71: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.768990] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-72: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769005] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-73: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769014] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-74: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769025] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-75: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769048] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-76: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769058] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-77: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769068] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-78: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769077] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-79: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769086] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-80: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769096] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-81: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769106] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-82: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769119] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-83: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769129] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-84: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769138] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-85: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769155] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-86: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769172] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-87: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769189] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-88: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769205] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-89: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769220] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-90: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769234] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-91: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769243] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-92: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769254] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-93: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769263] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-94: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769272] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-95: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769281] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-96: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769290] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-97: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769299] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-98: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769308] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-99: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769318] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-100: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769329] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-101: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769338] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-102: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769367] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-103: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769383] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-104: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769395] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-105: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769407] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-106: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769417] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-53: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:35.769433] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-107: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:36.016179] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.016754] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49219: ping timer event already removed >[2018-08-18 10:05:36.017177] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49219: ping timer event already removed >[2018-08-18 10:05:36.017536] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49219: ping timer event already removed >[2018-08-18 10:05:36.017821] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:36.018511] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7f3d0256d4e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7f3d0d44d631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7f3d02158820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.018557] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 10:05:36.018540] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:36.018556] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-31: ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b: generation now vs cached: 3, 3 >[2018-08-18 10:05:36.018851] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:36.019806] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.019860] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:36.019892] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:36.020225] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:36.021226] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:36.021451] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:36.021836] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.021850] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.021866] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:36.021886] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:36.021941] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:36.021958] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:36.022928] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.022978] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b): ordering.go is not set, hence not winding >[2018-08-18 10:05:36.022994] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 10:05:36.023246] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:36.023445] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:36.023624] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.023869] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7f3d0256c0b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7f3d0256ca5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.023897] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7f3d0256c0b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7f3d0256ca5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 10:05:36.023925] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): picking the request for winding >[2018-08-18 10:05:36.024132] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7f3d0256b6a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7f3d0256ca90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:36.024292] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:36.024312] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:36.024387] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.024431] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=2, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b): ordering.go is not set, hence not winding >[2018-08-18 10:05:36.024451] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 10:05:36.024759] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:36.024946] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:36.025240] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.025282] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): picking the request for winding >[2018-08-18 10:05:36.025290] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.025307] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): picking the request for winding >[2018-08-18 10:05:36.025335] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 10:05:36.025404] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:36.025429] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:36.025633] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:36.025849] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.026016] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:36.026196] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:36.026243] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:36.026384] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.026459] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 10:05:36.026410] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:36.026458] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:36.026665] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:36.026819] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): destroying request, removing from all queues >[2018-08-18 10:05:36.026963] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.026983] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=4): picking the request for winding >[2018-08-18 10:05:36.027310] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=4): destroying request, removing from all queues >[2018-08-18 10:05:36.027409] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:36.103604] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.103648] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:36.103701] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /7A/Tp/bSUB/my-bucket_0/79477581a2c811e8bffe246e96318264.t at vol0-replicate-14 >[2018-08-18 10:05:36.104070] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:36.104221] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /7A/Tp/bSUB/my-bucket_0/79477581a2c811e8bffe246e96318264.t returned with op_ret 0 >[2018-08-18 10:05:36.104820] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7f3d0256d4e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7f3d0d44d631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7f3d02158820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.104851] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:36.104883] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-14: f3c71c0d-9173-488b-933b-386804975422: generation now vs cached: 3, 3 >[2018-08-18 10:05:36.105143] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:36.105900] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.105937] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:36.105959] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:36.106272] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:36.107057] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:36.107289] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:36.107512] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.107540] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:36.107558] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:36.107635] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:36.107654] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:36.107644] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.108213] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.108244] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=f3c71c0d-9173-488b-933b-386804975422): ordering.go is not set, hence not winding >[2018-08-18 10:05:36.108262] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 10:05:36.108643] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.108861] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7f3d0256c0b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7f3d0256ca5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 10:05:36.108890] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=f3c71c0d-9173-488b-933b-386804975422): ordering.go is not set, hence not winding >[2018-08-18 10:05:36.109214] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:36.109452] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:36.109746] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7f3d0256b6a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7f3d0256ca90] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.109781] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7f3d0256b6a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7f3d0256ca90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:36.109823] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): picking the request for winding >[2018-08-18 10:05:36.109892] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:36.109912] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:36.110182] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.110423] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:36.110633] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:36.110678] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): picking the request for winding >[2018-08-18 10:05:36.110700] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 10:05:36.110752] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:36.110771] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:36.110863] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.111207] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.111337] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:36.111546] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:36.111588] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): picking the request for winding >[2018-08-18 10:05:36.111607] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 10:05:36.111662] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:36.111678] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:36.111743] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.112222] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.112287] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:36.112502] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): destroying request, removing from all queues >[2018-08-18 10:05:36.112545] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=4): picking the request for winding >[2018-08-18 10:05:36.112699] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:36.112869] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=4): destroying request, removing from all queues >[2018-08-18 10:05:36.113006] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:41.384487] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-42: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384529] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-43: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384540] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-0: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384550] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-1: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384558] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-2: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384572] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-3: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384581] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-4: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384590] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-5: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384598] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-6: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384607] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-7: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384616] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-8: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384625] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-9: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384634] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-10: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384652] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-11: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384661] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-12: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384670] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-13: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384679] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-14: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384688] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-15: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384696] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-16: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384707] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-17: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384717] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-18: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384727] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-19: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384737] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-20: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384747] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-21: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384756] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-22: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384765] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-23: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384774] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-24: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384783] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-25: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384791] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-26: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384799] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-27: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384808] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-28: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384818] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-29: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384827] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-30: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384835] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-31: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384844] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-32: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384852] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-33: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384863] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-34: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384877] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-35: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384887] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-36: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384897] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-37: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384907] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-38: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384916] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-39: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384924] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-40: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384933] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-41: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384943] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-45: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384952] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-46: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384961] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-47: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384970] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-48: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384979] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-49: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384987] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-50: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.384998] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-51: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385008] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-52: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385019] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-53: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385040] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-54: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385053] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-55: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385064] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-56: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385072] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-57: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385082] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-58: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385090] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-59: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385099] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-60: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385107] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-61: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385120] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-62: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385129] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-63: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385138] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-64: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385147] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-65: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385156] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-66: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385164] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-67: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385173] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-68: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385183] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-69: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385193] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-70: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385204] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-71: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385214] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-72: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385224] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-73: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385233] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-74: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385241] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-75: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385250] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-76: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385259] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-77: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385268] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-78: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385277] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-79: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385285] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-80: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385294] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-81: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385303] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-82: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385311] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-83: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385320] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-84: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385328] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-85: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385342] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-86: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385352] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-87: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385365] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-88: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385375] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-89: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385386] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-90: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385394] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-91: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385403] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-92: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385412] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-93: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385420] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-94: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385429] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-95: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385437] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-96: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385446] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-97: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385454] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-98: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385463] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-99: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385472] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-100: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385481] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-101: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385490] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-102: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385500] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-103: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385510] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-104: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385521] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-105: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385531] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-106: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385541] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-44: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:41.385550] D [rpc-clnt-ping.c:293:rpc_clnt_start_ping] 0-vol0-client-107: returning as transport is already disconnected OR there are no frames (0 || 0) >[2018-08-18 10:05:42.896217] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /SR on vol0-replicate-32 >[2018-08-18 10:05:42.896812] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /SR with op_ret 0 >[2018-08-18 10:05:42.907248] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /SR/Pn on vol0-replicate-33 >[2018-08-18 10:05:42.907746] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /SR/Pn with op_ret 0 >[2018-08-18 10:05:42.913059] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /SR/Pn/uao3 on vol0-replicate-32 >[2018-08-18 10:05:42.913466] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /SR/Pn/uao3 with op_ret 0 >[2018-08-18 10:05:42.917251] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /SR/Pn/uao3/mofunsky-img-sh1a_7 on vol0-replicate-31 >[2018-08-18 10:05:42.917679] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /SR/Pn/uao3/mofunsky-img-sh1a_7 with op_ret 0 >[2018-08-18 10:05:42.919805] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /SR/Pn/uao3/mofunsky-img-sh1a_7/b5cf5046a2a411e8b1a76c92bf6dcc7a.t on vol0-replicate-16 >[2018-08-18 10:05:42.920290] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /SR/Pn/uao3/mofunsky-img-sh1a_7/b5cf5046a2a411e8b1a76c92bf6dcc7a.t with op_ret 0 >[2018-08-18 10:05:42.920334] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-16 >[2018-08-18 10:05:42.920626] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-16: 801b33ff-1457-4b2d-9d44-cf90e1e4d256: generation now vs cached: 3, 3 >[2018-08-18 10:05:42.921627] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.921666] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=0): picking the request for winding >[2018-08-18 10:05:42.921691] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:42.921825] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /SR/Pn/uao3/mofunsky-img-sh1a_7/fb6eaeb8a22a11e8bffe246e96318264.t on vol0-replicate-7 >[2018-08-18 10:05:42.921968] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:42.922210] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /SR/Pn/uao3/mofunsky-img-sh1a_7/fb6eaeb8a22a11e8bffe246e96318264.t with op_ret 0 >[2018-08-18 10:05:42.922246] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-7 >[2018-08-18 10:05:42.922384] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.922420] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256): ordering.go is not set, hence not winding >[2018-08-18 10:05:42.922446] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 10:05:42.922839] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:42.923003] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:42.923197] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-7: 8bddf33d-7a82-45f3-8c98-9c08c1f773a3: generation now vs cached: 3, 3 >[2018-08-18 10:05:42.923242] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.923383] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.923451] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7f3d0256c0b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7f3d0256ca5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 10:05:42.923476] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=1): picking the request for winding >[2018-08-18 10:05:42.923633] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7f3d0256b6a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7f3d0256ca90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:42.923693] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:42.923709] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:42.923903] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.923934] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=2, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256): ordering.go is not set, hence not winding >[2018-08-18 10:05:42.923954] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 10:05:42.924373] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.924468] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.924483] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:42.924498] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=0): picking the request for winding >[2018-08-18 10:05:42.924518] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:42.924692] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:42.924756] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=2): picking the request for winding >[2018-08-18 10:05:42.924779] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=3): picking the request for winding >[2018-08-18 10:05:42.924794] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 10:05:42.924778] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:42.924854] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:42.924876] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:42.924931] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.925009] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:42.925108] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.925137] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3): ordering.go is not set, hence not winding >[2018-08-18 10:05:42.925153] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 10:05:42.925341] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:42.925564] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:42.925596] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:42.925756] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.925781] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=4, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256): ordering.go is not set, hence not winding >[2018-08-18 10:05:42.925788] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:42.925801] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=4): added req to liability queue. inode-generation-number=5 >[2018-08-18 10:05:42.925877] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:42.925936] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.926020] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.926105] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:42.926205] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7f3d0256c0b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7f3d0256ca5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77processing queues >[2018-08-18 10:05:42.926264] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7f3d0256c0b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7f3d0256ca5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 10:05:42.926289] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=1): picking the request for winding >[2018-08-18 10:05:42.926309] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=3): destroying request, removing from all queues >[2018-08-18 10:05:42.926381] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=4): picking the request for winding >[2018-08-18 10:05:42.926446] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-16: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:42.926457] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7f3d0256b6a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7f3d0256ca90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:42.926466] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:42.926493] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.926547] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:42.926564] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 10:05:42.926702] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.926727] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=2, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3): ordering.go is not set, hence not winding >[2018-08-18 10:05:42.926743] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 10:05:42.926987] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=4): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:42.927077] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.927158] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.927257] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:42.927269] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=4): destroying request, removing from all queues >[2018-08-18 10:05:42.927312] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=5): picking the request for winding >[2018-08-18 10:05:42.927500] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:42.927514] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.927564] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=2): picking the request for winding >[2018-08-18 10:05:42.927597] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=3): picking the request for winding >[2018-08-18 10:05:42.927609] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 10:05:42.927668] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:42.927687] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 10:05:42.927734] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=801b33ff-1457-4b2d-9d44-cf90e1e4d256, gen=5): destroying request, removing from all queues >[2018-08-18 10:05:42.927749] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.927804] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:42.927868] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-16: lk op is for a transaction >[2018-08-18 10:05:42.928224] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:42.928260] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.928453] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:42.928492] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=4, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3): ordering.go is not set, hence not winding >[2018-08-18 10:05:42.928507] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=4): added req to liability queue. inode-generation-number=5 >[2018-08-18 10:05:42.928674] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.928704] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=4): picking the request for winding >[2018-08-18 10:05:42.928760] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=5, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:42.928809] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-7: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:42.928827] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 10:05:42.928868] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.928900] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=5, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:42.929220] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:42.929420] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=3): destroying request, removing from all queues >[2018-08-18 10:05:42.929609] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.929640] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=5, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3. CONFLICT: unique=0, fop=WRITE, gen=4, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:42.929715] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 10:05:42.929933] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=4): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:42.930138] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=4): destroying request, removing from all queues >[2018-08-18 10:05:42.930326] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:42.930352] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=5): picking the request for winding >[2018-08-18 10:05:42.930694] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=8bddf33d-7a82-45f3-8c98-9c08c1f773a3, gen=5): destroying request, removing from all queues >[2018-08-18 10:05:42.930826] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-7: lk op is for a transaction >[2018-08-18 10:05:49.899516] D [MSGID: 0] [dht-common.c:2743:dht_lookup] 0-vol0-dht: Calling fresh lookup for /vL/jp/KnUC/cpsbxt_5/b1c54979a2cc11e8bae8246e96317804.t on vol0-replicate-24 >[2018-08-18 10:05:49.900110] D [MSGID: 0] [dht-common.c:2328:dht_lookup_cbk] 0-vol0-dht: fresh_lookup returned for /vL/jp/KnUC/cpsbxt_5/b1c54979a2cc11e8bae8246e96317804.t with op_ret 0 >[2018-08-18 10:05:49.900160] D [MSGID: 0] [dht-layout.c:870:dht_layout_preset] 0-vol0-dht: file = 00000000-0000-0000-0000-000000000000, subvol = vol0-replicate-24 >[2018-08-18 10:05:49.900496] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-24: 6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd: generation now vs cached: 3, 3 >[2018-08-18 10:05:49.901538] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:49.901574] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=0): picking the request for winding >[2018-08-18 10:05:49.901596] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:49.901867] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:49.902660] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:49.902864] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:49.903078] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:49.903105] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=0): picking the request for winding >[2018-08-18 10:05:49.903125] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:49.903187] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:49.903203] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 10:05:49.903319] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:49.903904] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:49.903938] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd): ordering.go is not set, hence not winding >[2018-08-18 10:05:49.903958] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 10:05:49.904426] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:49.904721] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:49.904940] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:49.905176] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:49.905300] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7f3d0256c0b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7f3d0256ca5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 10:05:49.905337] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=1): picking the request for winding >[2018-08-18 10:05:49.905563] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7f3d0256b6a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7f3d0256ca90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:49.905630] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:49.905651] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 10:05:49.906252] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:49.906299] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:49.906522] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:49.906603] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=2): picking the request for winding >[2018-08-18 10:05:49.906635] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 10:05:49.906699] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:49.906708] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:49.906717] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 10:05:49.907198] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:49.907309] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:49.907388] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:49.907426] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=3): picking the request for winding >[2018-08-18 10:05:49.907447] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 10:05:49.907498] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-24: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:49.907515] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 10:05:49.907557] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:49.908112] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:49.908155] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:49.908370] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=3): destroying request, removing from all queues >[2018-08-18 10:05:49.908420] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=4): picking the request for winding >[2018-08-18 10:05:49.908581] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:49.908783] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=6ebbf3b8-7ba2-43c9-84a5-a47d97be10cd, gen=4): destroying request, removing from all queues >[2018-08-18 10:05:49.908917] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-24: lk op is for a transaction >[2018-08-18 10:05:54.692333] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.693018] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:54.694353] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7f3d0256d4e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7f3d0d44d631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7f3d02158820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.694395] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >The message "D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/sjkzhdd_1/8d68fb4ea2cb11e88a876c92bf6dccae.t returned with op_ret 0" repeated 2 times between [2018-08-18 10:04:41.107020] and [2018-08-18 10:05:54.693251] >[2018-08-18 10:05:54.694393] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:54.694469] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-31: ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b: generation now vs cached: 3, 3 >[2018-08-18 10:05:54.694781] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:54.695659] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.695701] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:54.695731] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:54.696053] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:54.696825] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:54.697051] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:54.697319] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.697348] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:54.697393] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:54.697415] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.697522] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:54.697961] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.697990] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 10:05:54.697543] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:54.697989] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b): ordering.go is not set, hence not winding >[2018-08-18 10:05:54.698028] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 10:05:54.698439] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.699285] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7f3d0256c0b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7f3d0256ca5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 10:05:54.699313] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b): ordering.go is not set, hence not winding >[2018-08-18 10:05:54.699576] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:54.699784] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:54.699998] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.700085] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7f3d0256b6a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7f3d0256ca90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:54.700136] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): picking the request for winding >[2018-08-18 10:05:54.700285] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 10:05:54.700260] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:54.700276] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:54.700461] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.700488] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=2, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b): ordering.go is not set, hence not winding >[2018-08-18 10:05:54.700507] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 10:05:54.700822] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:54.701023] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:54.701225] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.701251] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): picking the request for winding >[2018-08-18 10:05:54.701269] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): picking the request for winding >[2018-08-18 10:05:54.701282] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 10:05:54.701338] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:54.701354] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:54.701359] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.701498] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:54.701776] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.701849] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:54.702040] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:54.702121] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:54.702224] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.702303] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 10:05:54.702253] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:54.702302] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:54.702515] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:54.702682] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): destroying request, removing from all queues >[2018-08-18 10:05:54.702843] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.702868] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=4): picking the request for winding >[2018-08-18 10:05:54.703164] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=4): destroying request, removing from all queues >[2018-08-18 10:05:54.703273] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:54.872710] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.873146] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:54.873223] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >The message "D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/sjkzhdd_1/8d68fb4ea2cb11e88a876c92bf6dccae.t at vol0-replicate-31" repeated 3 times between [2018-08-18 10:04:41.106536] and [2018-08-18 10:05:54.872779] >[2018-08-18 10:05:54.873222] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/sjkzhdd_1/8d68fb4ea2cb11e88a876c92bf6dccae.t returned with op_ret 0 >[2018-08-18 10:05:54.873775] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7f3d0256d4e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7f3d0d44d631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7f3d02158820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.873807] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:54.873841] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-31: ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b: generation now vs cached: 3, 3 >[2018-08-18 10:05:54.874118] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:54.874833] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.874866] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:54.874888] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:54.875154] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:54.875943] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:54.876191] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:54.876395] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.877179] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.877219] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:54.877240] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:54.877298] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:54.878312] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.878346] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 10:05:54.877315] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:54.878345] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b): ordering.go is not set, hence not winding >[2018-08-18 10:05:54.878380] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 10:05:54.878783] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.878983] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7f3d0256c0b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7f3d0256ca5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 10:05:54.879009] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b): ordering.go is not set, hence not winding >[2018-08-18 10:05:54.879267] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:54.879460] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:54.879620] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.879760] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7f3d0256b6a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7f3d0256ca90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:54.879897] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 10:05:54.879892] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:54.879897] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:54.880175] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.880368] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:54.880559] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:54.880636] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): picking the request for winding >[2018-08-18 10:05:54.880669] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 10:05:54.880727] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:54.880751] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:54.880750] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.881241] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.881334] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:54.881538] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:54.881595] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): picking the request for winding >[2018-08-18 10:05:54.881616] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 10:05:54.881677] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:54.881693] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:54.881728] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.882228] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:54.882280] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.882427] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=3): destroying request, removing from all queues >[2018-08-18 10:05:54.882465] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=4): picking the request for winding >[2018-08-18 10:05:54.882671] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.882779] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=4): destroying request, removing from all queues >[2018-08-18 10:05:54.882908] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:54.892698] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.892759] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >The message "D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding" repeated 4 times between [2018-08-18 10:04:41.106489] and [2018-08-18 10:05:54.892737] >[2018-08-18 10:05:54.892758] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/sjkzhdd_1/8d68fb4ea2cb11e88a876c92bf6dccae.t at vol0-replicate-31 >[2018-08-18 10:05:54.893115] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:54.893159] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/sjkzhdd_1/8d68fb4ea2cb11e88a876c92bf6dccae.t returned with op_ret 0 >[2018-08-18 10:05:54.893627] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7f3d0256d4e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7f3d0d44d631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7f3d02158820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.893656] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:54.893685] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-31: ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b: generation now vs cached: 3, 3 >[2018-08-18 10:05:54.893913] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:54.894360] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7f3d0256d4e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7f3d0d44d631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7f3d02158820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.894389] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:54.894422] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-31: ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b: generation now vs cached: 3, 3 >[2018-08-18 10:05:54.894646] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:54.895260] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.895293] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:54.895311] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:54.895531] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-31: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:54.895705] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.895743] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=1, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, conflicts-sync-failed?=yes, conflicts-error=Success >[2018-08-18 10:05:54.896125] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:54.896314] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:54.896486] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.896514] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): picking the request for winding >[2018-08-18 10:05:54.896809] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:54.921909] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.921945] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk >[2018-08-18 10:05:54.896914] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-31: lk op is for a transaction >[2018-08-18 10:05:54.921944] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:54.921998] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /7A/Tp/bSUB/my-bucket_0/79477581a2c811e8bffe246e96318264.t at vol0-replicate-14 >[2018-08-18 10:05:54.922404] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.17:49202: ping timer event already removed >[2018-08-18 10:05:54.922717] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.27:49202: ping timer event already removed >[2018-08-18 10:05:54.922973] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7f3d0d8d0d0b] (--> /lib64/libgfrpc.so.0(+0x14331)[0x7f3d0d8d1331] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x451)[0x7f3d0d8ce0f1] (--> /usr/lib64/glusterfs/3.10.12/xlator/protocol/client.so(+0x10342)[0x7f3d02c98342] ))))) 0-: 10.16.71.37:49202: ping timer event already removed >[2018-08-18 10:05:54.923124] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /7A/Tp/bSUB/my-bucket_0/79477581a2c811e8bffe246e96318264.t returned with op_ret 0 >[2018-08-18 10:05:54.923234] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:54.923730] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7f3d0256d4e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7f3d0d44d631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7f3d02158820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.923760] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:54.923790] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-14: f3c71c0d-9173-488b-933b-386804975422: generation now vs cached: 3, 3 >[2018-08-18 10:05:54.924022] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:54.924844] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.924891] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:54.924915] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:54.925233] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:54.926102] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:54.926355] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:54.926591] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.927191] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.927224] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:54.927244] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:54.927295] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:54.927311] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:54.928194] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.928225] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=f3c71c0d-9173-488b-933b-386804975422): ordering.go is not set, hence not winding >[2018-08-18 10:05:54.928243] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): added req to liability queue. inode-generation-number=2 >[2018-08-18 10:05:54.928626] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.928867] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x80b3)[0x7f3d0256c0b3] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a5d)[0x7f3d0256ca5d] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): request fulfilled. removing the request from liability queue? = no >[2018-08-18 10:05:54.928896] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=1, gfid=f3c71c0d-9173-488b-933b-386804975422): ordering.go is not set, hence not winding >[2018-08-18 10:05:54.929099] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x76a8)[0x7f3d0256b6a8] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a90)[0x7f3d0256ca90] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:54.929368] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:54.929566] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:54.929904] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.929920] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.929932] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): picking the request for winding >[2018-08-18 10:05:54.929961] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): picking the request for winding >[2018-08-18 10:05:54.929975] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): added req to liability queue. inode-generation-number=3 >[2018-08-18 10:05:54.930039] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:54.930059] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:54.930179] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:54.930552] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.930679] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:54.930916] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:54.930962] D [MSGID: 0] [write-behind.c:1605:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gen=3, gfid=f3c71c0d-9173-488b-933b-386804975422): ordering.go is not set, hence not winding >[2018-08-18 10:05:54.930983] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): added req to liability queue. inode-generation-number=4 >[2018-08-18 10:05:54.931150] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.931189] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): picking the request for winding >[2018-08-18 10:05:54.931219] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=f3c71c0d-9173-488b-933b-386804975422. CONFLICT: unique=0, fop=WRITE, gen=2, gfid=f3c71c0d-9173-488b-933b-386804975422, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:54.931273] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:54.931292] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:54.931392] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.931427] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=f3c71c0d-9173-488b-933b-386804975422. CONFLICT: unique=0, fop=WRITE, gen=2, gfid=f3c71c0d-9173-488b-933b-386804975422, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:54.931696] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:54.931897] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=2): destroying request, removing from all queues >[2018-08-18 10:05:54.932102] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.932138] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=4, gfid=f3c71c0d-9173-488b-933b-386804975422. CONFLICT: unique=0, fop=WRITE, gen=3, gfid=f3c71c0d-9173-488b-933b-386804975422, conflicts-sync-failed?=no, conflicts-error=Success >[2018-08-18 10:05:54.932213] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:54.932436] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:54.932624] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=3): destroying request, removing from all queues >[2018-08-18 10:05:54.932809] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.932835] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=4): picking the request for winding >[2018-08-18 10:05:54.933238] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=4): destroying request, removing from all queues >[2018-08-18 10:05:54.933382] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:54.942532] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0xb880)[0x7f3d0256f880] (--> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f3d0d44e3bd] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x2596)[0x7f3d02156596] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.942567] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:54.942605] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /7A/Tp/bSUB/my-bucket_0/79477581a2c811e8bffe246e96318264.t at vol0-replicate-14 >[2018-08-18 10:05:54.942920] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=LOOKUP, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:54.943024] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /7A/Tp/bSUB/my-bucket_0/79477581a2c811e8bffe246e96318264.t returned with op_ret 0 >[2018-08-18 10:05:54.943557] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7f3d0256d4e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7f3d0d44d631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7f3d02158820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.943587] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:54.943620] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-14: f3c71c0d-9173-488b-933b-386804975422: generation now vs cached: 3, 3 >[2018-08-18 10:05:54.943856] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:54.944205] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x94e8)[0x7f3d0256d4e8] (--> /lib64/libglusterfs.so.0(default_readv+0xe1)[0x7f3d0d44d631] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x4820)[0x7f3d02158820] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.944234] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=READ, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:54.944263] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-vol0-replicate-14: f3c71c0d-9173-488b-933b-386804975422: generation now vs cached: 3, 3 >[2018-08-18 10:05:54.944486] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=READ, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:54.945119] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9224)[0x7f3d0256d224] (--> /lib64/libglusterfs.so.0(default_writev+0xf9)[0x7f3d0d44e1e9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/quick-read.so(+0x3173)[0x7f3d02157173] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.945162] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): picking the request for winding >[2018-08-18 10:05:54.945186] D [MSGID: 0] [write-behind.c:1250:__wb_pick_unwinds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): added req to liability queue. inode-generation-number=1 >[2018-08-18 10:05:54.945407] D [MSGID: 0] [afr-transaction.c:1883:afr_post_nonblocking_inodelk_cbk] 0-vol0-replicate-14: Non blocking inodelks done. Proceeding to FOP >[2018-08-18 10:05:54.945609] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x9938)[0x7f3d0256d938] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] (--> /lib64/libglusterfs.so.0(default_flush+0xc1)[0x7f3d0d44d161] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.945646] D [MSGID: 0] [write-behind.c:1578:__wb_pick_winds] 0-vol0-write-behind: Not winding request due to a conflicting write in liability queue. REQ: unique=0, fop=FLUSH, gen=1, gfid=f3c71c0d-9173-488b-933b-386804975422. CONFLICT: unique=0, fop=WRITE, gen=0, gfid=f3c71c0d-9173-488b-933b-386804975422, conflicts-sync-failed?=yes, conflicts-error=Success >[2018-08-18 10:05:54.946132] D [write-behind.c:715:__wb_fulfill_request] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6827)[0x7f3d0256a827] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] ))))) 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): request fulfilled. removing the request from liability queue? = yes >[2018-08-18 10:05:54.946391] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6857)[0x7f3d0256a857] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x6b88)[0x7f3d0256ab88] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b6f)[0x7f3d0256cb6f] ))))) 0-vol0-write-behind: (unique = 0, fop=WRITE, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=0): destroying request, removing from all queues >[2018-08-18 10:05:54.946619] D [write-behind.c:1696:wb_process_queue] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8adb)[0x7f3d0256cadb] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8b77)[0x7f3d0256cb77] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/distribute.so(+0x604cd)[0x7f3d027d84cd] (--> /usr/lib64/glusterfs/3.10.12/xlator/cluster/replicate.so(+0x22812)[0x7f3d02a2c812] ))))) 0-vol0-write-behind: processing queues >[2018-08-18 10:05:54.946653] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=FLUSH, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): picking the request for winding >[2018-08-18 10:05:54.947136] D [write-behind.c:380:__wb_request_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f3d0d3d4fab] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x2582)[0x7f3d02566582] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x46d9)[0x7f3d025686d9] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8991)[0x7f3d0256c991] (--> /usr/lib64/glusterfs/3.10.12/xlator/performance/write-behind.so(+0x8a9b)[0x7f3d0256ca9b] ))))) 0-vol0-write-behind: (unique = 0, fop=FLUSH, gfid=f3c71c0d-9173-488b-933b-386804975422, gen=1): destroying request, removing from all queues >[2018-08-18 10:05:54.947304] D [MSGID: 0] [afr-lk-common.c:448:transaction_lk_op] 0-vol0-replicate-14: lk op is for a transaction >[2018-08-18 10:05:11.045928] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=LOOKUP, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=0): picking the request for winding >[2018-08-18 10:05:11.045989] D [MSGID: 0] [dht-common.c:2668:dht_lookup] 0-vol0-dht: calling revalidate lookup for /vL/jp/KnUC/sjkzhdd_1/8d68fb4ea2cb11e88a876c92bf6dccae.t at vol0-replicate-31 >[2018-08-18 10:05:11.047400] D [MSGID: 0] [dht-common.c:1046:dht_revalidate_cbk] 0-vol0-dht: revalidate lookup of /vL/jp/KnUC/sjkzhdd_1/8d68fb4ea2cb11e88a876c92bf6dccae.t returned with op_ret 0 >[2018-08-18 10:05:54.879805] D [MSGID: 0] [write-behind.c:1646:__wb_pick_winds] 0-vol0-write-behind: (unique=0, fop=WRITE, gfid=ae6fc2f7-f0c3-46c8-8546-9e8b48612c1b, gen=1): picking the request for winding
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1618925
: 1476753