Bug 1297004 - [write-behind] : Write/Append to a full volume causes fuse client to crash
Summary: [write-behind] : Write/Append to a full volume causes fuse client to crash
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: distribute
Version: rhgs-3.1
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: RHGS 3.1.2
Assignee: Bug Updates Notification Mailing List
QA Contact: krishnaram Karthick
URL:
Whiteboard:
Depends On:
Blocks: 1297373
TreeView+ depends on / blocked
 
Reported: 2016-01-08 17:07 UTC by Joseph Elwin Fernandes
Modified: 2016-03-01 06:07 UTC (History)
6 users (show)

Fixed In Version: glusterfs-3.7.5-16
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1297373 (view as bug list)
Environment:
Last Closed: 2016-03-01 06:07:03 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:0193 0 normal SHIPPED_LIVE Red Hat Gluster Storage 3.1 update 2 2016-03-01 10:20:36 UTC

Description Joseph Elwin Fernandes 2016-01-08 17:07:12 UTC
Description of problem:
Write/Append to a full volume causes fuse client to crash 

Steps to Reproduce:
1. Create a 1x2 volume 
[root@fedora2 /]# gluster vol info
 
Volume Name: test
Type: Replicate
Volume ID: 538b186b-b213-49f5-a5d9-458a3ba2690a
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: fedora2:/home/ssd/small_brick0/s0
Brick2: fedora2:/home/ssd/small_brick1/s1
Options Reconfigured:
performance.readdir-ahead: on
[root@fedora2 /]# 


with each brick to be 1 GB

2. on a fuse mount issue this command

for i in {1..1024}; do cat /var/log/journal/49001fda54c6e148ba137302d159e716/system >> file1; done

The input file is of 8MB and soon the volume will be full.

3. You get NO Disk space error and then the fuse client crashes.

(gdb) bt
#0  0x00007f627da49d4e in wb_do_unwinds (wb_inode=0x7f626c004170, lies=0x7f62776fc7a0) at write-behind.c:1069
#1  0x00007f627da4aae5 in wb_process_queue (wb_inode=0x7f626c004170) at write-behind.c:1453
#2  0x00007f627da493ad in wb_fulfill_cbk (frame=0x7f626800c0ec, cookie=0x7f626801150c, this=0x7f627800d970, 
    op_ret=-1, op_errno=28, prebuf=0x7f6268026964, postbuf=0x7f62680269d4, xdata=0x0) at write-behind.c:920
#3  0x00007f627dcacb83 in dht_writev_cbk (frame=0x7f626801150c, cookie=0x7f626801194c, this=0x7f627800c540, 
    op_ret=-1, op_errno=28, prebuf=0x7f6268026964, postbuf=0x7f62680269d4, xdata=0x0) at dht-inode-write.c:106
#4  0x00007f627def5b42 in afr_writev_unwind (frame=0x7f626801194c, this=0x7f627800af50) at afr-inode-write.c:204
#5  0x00007f627def5bd7 in afr_transaction_writev_unwind (frame=0x7f6268011bdc, this=0x7f627800af50)
    at afr-inode-write.c:221
#6  0x00007f627df044a6 in __afr_txn_write_done (frame=0x7f6268011bdc, this=0x7f627800af50) at afr-transaction.c:85
#7  0x00007f627df0aaa5 in afr_unlock_common_cbk (frame=0x7f6268011bdc, cookie=0x1, this=0x7f627800af50, op_ret=0, 
    op_errno=0, xdata=0x0) at afr-lk-common.c:633
#8  0x00007f627df0acb2 in afr_unlock_inodelk_cbk (frame=0x7f6268011bdc, cookie=0x1, this=0x7f627800af50, op_ret=0, 
    op_errno=0, xdata=0x0) at afr-lk-common.c:674
#9  0x00007f627e16cc0d in client3_3_finodelk_cbk (req=0x7f626000129c, iov=0x7f62600012dc, count=1, 
    myframe=0x7f626000119c) at client-rpc-fops.c:1676
#10 0x00007f628b22c1af in rpc_clnt_handle_reply (clnt=0x7f6278033c10, pollin=0x7f626801b690) at rpc-clnt.c:759
#11 0x00007f628b22c625 in rpc_clnt_notify (trans=0x7f62780340a0, mydata=0x7f6278033c40, 
    event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7f626801b690) at rpc-clnt.c:900
#12 0x00007f628b228a1b in rpc_transport_notify (this=0x7f62780340a0, event=RPC_TRANSPORT_MSG_RECEIVED, 
    data=0x7f626801b690) at rpc-transport.c:541
#13 0x00007f628064cb3c in socket_event_poll_in (this=0x7f62780340a0) at socket.c:2231
#14 0x00007f628064d06a in socket_event_handler (fd=11, idx=1, data=0x7f62780340a0, poll_in=1, poll_out=0, poll_err=0)
    at socket.c:2344
#15 0x00007f628b4c0a24 in event_dispatch_epoll_handler (event_pool=0x1897d70, event=0x7f62776fcea0)
    at event-epoll.c:571
#16 0x00007f628b4c0dc6 in event_dispatch_epoll_worker (data=0x7f6278033800) at event-epoll.c:674
#17 0x00007f628a2b752a in start_thread () from /lib64/libpthread.so.0
#18 0x00007f6289c0622d in clone () from /lib64/libc.so.6
(gdb) 


Actual results:

Fuse client shouldnt crash

Expected results:

Fuse client crashes

Additional info:

Always reproducible.

Comment 5 krishnaram Karthick 2016-01-30 07:19:11 UTC
Verified in build glusterfs-server-3.7.5-17.el7rhgs.x86_64

Appended a file to exceed hot tier's capacity. Once the file exceeded hot tier's capacity, writes failed as expected but no crashes were seen from the client and no IO hangs were observed.

IO's failed with the following error msg,

dd: error writing ‘largefile’: No space left on device
dd: closing output file ‘largefile’: No space left on device

Comment 7 errata-xmlrpc 2016-03-01 06:07:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0193.html


Note You need to log in before you can comment on or make changes to this bug.