Description of problem: Write/Append to a full volume causes fuse client to crash Steps to Reproduce: 1. Create a 1x2 volume [root@fedora2 /]# gluster vol info Volume Name: test Type: Replicate Volume ID: 538b186b-b213-49f5-a5d9-458a3ba2690a Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: fedora2:/home/ssd/small_brick0/s0 Brick2: fedora2:/home/ssd/small_brick1/s1 Options Reconfigured: performance.readdir-ahead: on [root@fedora2 /]# with each brick to be 1 GB 2. on a fuse mount issue this command for i in {1..1024}; do cat /var/log/journal/49001fda54c6e148ba137302d159e716/system >> file1; done The input file is of 8MB and soon the volume will be full. 3. You get NO Disk space error and then the fuse client crashes. (gdb) bt #0 0x00007f627da49d4e in wb_do_unwinds (wb_inode=0x7f626c004170, lies=0x7f62776fc7a0) at write-behind.c:1069 #1 0x00007f627da4aae5 in wb_process_queue (wb_inode=0x7f626c004170) at write-behind.c:1453 #2 0x00007f627da493ad in wb_fulfill_cbk (frame=0x7f626800c0ec, cookie=0x7f626801150c, this=0x7f627800d970, op_ret=-1, op_errno=28, prebuf=0x7f6268026964, postbuf=0x7f62680269d4, xdata=0x0) at write-behind.c:920 #3 0x00007f627dcacb83 in dht_writev_cbk (frame=0x7f626801150c, cookie=0x7f626801194c, this=0x7f627800c540, op_ret=-1, op_errno=28, prebuf=0x7f6268026964, postbuf=0x7f62680269d4, xdata=0x0) at dht-inode-write.c:106 #4 0x00007f627def5b42 in afr_writev_unwind (frame=0x7f626801194c, this=0x7f627800af50) at afr-inode-write.c:204 #5 0x00007f627def5bd7 in afr_transaction_writev_unwind (frame=0x7f6268011bdc, this=0x7f627800af50) at afr-inode-write.c:221 #6 0x00007f627df044a6 in __afr_txn_write_done (frame=0x7f6268011bdc, this=0x7f627800af50) at afr-transaction.c:85 #7 0x00007f627df0aaa5 in afr_unlock_common_cbk (frame=0x7f6268011bdc, cookie=0x1, this=0x7f627800af50, op_ret=0, op_errno=0, xdata=0x0) at afr-lk-common.c:633 #8 0x00007f627df0acb2 in afr_unlock_inodelk_cbk (frame=0x7f6268011bdc, cookie=0x1, this=0x7f627800af50, op_ret=0, op_errno=0, xdata=0x0) at afr-lk-common.c:674 #9 0x00007f627e16cc0d in client3_3_finodelk_cbk (req=0x7f626000129c, iov=0x7f62600012dc, count=1, myframe=0x7f626000119c) at client-rpc-fops.c:1676 #10 0x00007f628b22c1af in rpc_clnt_handle_reply (clnt=0x7f6278033c10, pollin=0x7f626801b690) at rpc-clnt.c:759 #11 0x00007f628b22c625 in rpc_clnt_notify (trans=0x7f62780340a0, mydata=0x7f6278033c40, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7f626801b690) at rpc-clnt.c:900 #12 0x00007f628b228a1b in rpc_transport_notify (this=0x7f62780340a0, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7f626801b690) at rpc-transport.c:541 #13 0x00007f628064cb3c in socket_event_poll_in (this=0x7f62780340a0) at socket.c:2231 #14 0x00007f628064d06a in socket_event_handler (fd=11, idx=1, data=0x7f62780340a0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2344 #15 0x00007f628b4c0a24 in event_dispatch_epoll_handler (event_pool=0x1897d70, event=0x7f62776fcea0) at event-epoll.c:571 #16 0x00007f628b4c0dc6 in event_dispatch_epoll_worker (data=0x7f6278033800) at event-epoll.c:674 #17 0x00007f628a2b752a in start_thread () from /lib64/libpthread.so.0 #18 0x00007f6289c0622d in clone () from /lib64/libc.so.6 (gdb) Actual results: Fuse client shouldnt crash Expected results: Fuse client crashes Additional info: Always reproducible.
https://code.engineering.redhat.com/gerrit/#/c/65456/
Verified in build glusterfs-server-3.7.5-17.el7rhgs.x86_64 Appended a file to exceed hot tier's capacity. Once the file exceeded hot tier's capacity, writes failed as expected but no crashes were seen from the client and no IO hangs were observed. IO's failed with the following error msg, dd: error writing ‘largefile’: No space left on device dd: closing output file ‘largefile’: No space left on device
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0193.html