Hide Forgot
With below test-case, marker accounting becomes bad: 1) Create a volume with 1 brick 2) fuse mount 3) on one terminal write some data dd if=/dev/zero of=f1 bs=1M count=500 oflag=sync 4) on another terminal execute below rename operation while the write is still in progress for i in {1..50}; do ii=`expr $i + 1`; mv f$i f$ii; done
Enable quota on volume vol0 and set limit-usage to 1 GB of root of the volume [root@darkknightrises tmp]# gluster v quota vol0 list Path Hard-limit Soft-limit Used Available Soft-limit exceeded? Hard-limit exceeded? --------------------------------------------------------------------------------------------------------------------------- / 1.0GB 80% 0Bytes 1.0GB No Tried running command from one terminal dd if=/dev/zero of=f1 bs=1M count=500 oflag=sync On another terminal execute below rename operation while the write is still in progress for i in {1..50}; do ii=`expr $i + 1`; mv f$i f$ii; brick process crash on one replica pair storage node. bt from one storage node ================================= (gdb) bt #0 0x00007f5da09bb387 in _quota_enforcer_lookup (data=0x7f5dae24eca4) at quota-enforcer-client.c:259 #1 0x00007f5da09bb4ce in quota_enforcer_lookup (frame=0x7f5dae24eca4, this=<value optimized out>, xdata=<value optimized out>, validate_cbk=<value optimized out>) at quota-enforcer-client.c:344 #2 0x00007f5da09af5cc in quota_validate (frame=0x7f5dae24eca4, inode=<value optimized out>, this=0x7f5d9c019670, cbk_fn=0x7f5da09b9590 <quota_validate_cbk>) at quota.c:901 #3 0x00007f5da09af8ce in quota_check_size_limit (frame=0x7f5dae24eca4, ctx=0x7f5d74002660, priv=0x7f5d9c037ea0, _inode=0x7f5d88e0906c, this=0x7f5d9c019670, op_errno=0x7f5d886843dc, just_validated=0, delta=131072, local=0x7f5d9c038298, skip_check=0x7f5d886843d8) at quota.c:1080 #4 0x00007f5da09b72a2 in quota_check_limit (frame=0x7f5dae24eca4, inode=<value optimized out>, this=0x7f5d9c019670, name=0x0, par=<value optimized out>) at quota.c:1204 #5 0x00007f5da09b9480 in quota_check_limit_continuation (parents=0x7f5d88684500, inode=0x7f5d88e09108, op_ret=<value optimized out>, op_errno=0, data=0x7f5dae24eca4) at quota.c:959 #6 0x00007f5da09b29f3 in quota_build_ancestry_cbk (frame=0x7f5dae085760, cookie=<value optimized out>, this=0x7f5d9c019670, op_ret=<value optimized out>, op_errno=0, entries=<value optimized out>, xdata=0x0) at quota.c:742 #7 0x00007f5da0bd580a in marker_build_ancestry_cbk (frame=<value optimized out>, cookie=<value optimized out>, this=<value optimized out>, op_ret=2, op_errno=0, entries=0x7f5d886849f0, xdata=0x0) at marker.c:2791 #8 0x00007f5db06a0202 in default_readdirp_cbk (frame=0x7f5dae24ed50, cookie=<value optimized out>, this=<value optimized out>, op_ret=2, op_errno=0, entries=<value optimized out>, xdata=0x0) at defaults.c:1237 #9 0x00007f5da1415860 in up_readdir_cbk (frame=0x7f5dae24eea8, cookie=<value optimized out>, this=<value optimized out>, op_ret=2, op_errno=0, entries=0x7f5d886849f0, xdata=0x0) at upcall.c:1292 #10 0x00007f5da1629555 in pl_readdirp_cbk (frame=0x7f5dae24fb6c, cookie=<value optimized out>, this=<value optimized out>, op_ret=2, op_errno=0, entries=0x7f5d886849f0, xdata=0x0) at posix.c:2202 #11 0x00007f5da1841e7d in posix_acl_readdirp_cbk (frame=0x7f5dae24e89c, cookie=<value optimized out>, this=<value optimized out>, op_ret=2, op_errno=0, entries=0x7f5d886849f0, xdata=0x0) at posix-acl.c:1591 ---Type <return> to continue, or q <return> to quit--- #12 0x00007f5da1a5396c in br_stub_readdirp_cbk (frame=0x7f5dae24d9d4, cookie=<value optimized out>, this=0x7f5da1841c00, op_ret=2, op_errno=0, entries=0x7f5d886849f0, dict=0x0) at bit-rot-stub.c:2249 #13 0x00007f5da2f85c83 in posix_readdirp (frame=0x7f5dae24da80, this=<value optimized out>, fd=<value optimized out>, size=<value optimized out>, off=<value optimized out>, dict=<value optimized out>) at posix.c:5580 #14 0x00007f5db0699633 in default_readdirp (frame=0x7f5dae24da80, this=0x7f5d9c0092a0, fd=0x7f5d9c0bcce4, size=0, off=<value optimized out>, xdata=<value optimized out>) at defaults.c:2101 #15 0x00007f5db0699633 in default_readdirp (frame=0x7f5dae24da80, this=0x7f5d9c00a860, fd=0x7f5d9c0bcce4, size=0, off=<value optimized out>, xdata=<value optimized out>) at defaults.c:2101 #16 0x00007f5db0699633 in default_readdirp (frame=0x7f5dae24da80, this=0x7f5d9c00d0f0, fd=0x7f5d9c0bcce4, size=0, off=<value optimized out>, xdata=<value optimized out>) at defaults.c:2101 #17 0x00007f5da1a4ba44 in br_stub_readdirp (frame=0x7f5dae24d9d4, this=0x7f5d9c00f000, fd=0x7f5d9c0bcce4, size=0, offset=0, dict=0x7f5dadc4ccb4) at bit-rot-stub.c:2284 #18 0x00007f5da183e7ad in posix_acl_readdirp (frame=0x7f5dae24e89c, this=0x7f5d9c010490, fd=0x7f5d9c0bcce4, size=0, offset=0, dict=<value optimized out>) at posix-acl.c:1625 #19 0x00007f5da162a755 in pl_readdirp (frame=0x7f5dae24fb6c, this=0x7f5d9c011820, fd=0x7f5d9c0bcce4, size=0, offset=0, xdata=0x7f5dadc4ccb4) at posix.c:2213 #20 0x00007f5da1410ea7 in up_readdirp (frame=0x7f5dae24eea8, this=0x7f5d9c012ba0, fd=0x7f5d9c0bcce4, size=0, off=0, dict=0x7f5dadc4ccb4) at upcall.c:1342 #21 0x00007f5db069bf43 in default_readdirp_resume (frame=0x7f5dae24ed50, this=0x7f5d9c014000, fd=0x7f5d9c0bcce4, size=0, off=0, xdata=0x7f5dadc4ccb4) at defaults.c:1657 #22 0x00007f5db06bc580 in call_resume (stub=0x7f5dadcda60c) at call-stub.c:2576 #23 0x00007f5da1208541 in iot_worker (data=0x7f5d9c053f70) at io-threads.c:215 #24 0x00007f5daf77ca51 in ?? () #25 0x00007f5d88685700 in ?? () #26 0x0000000000000000 in ?? () ===================================================== bt from another storage node ====================================================== Program terminated with signal 11, Segmentation fault. #0 0x00007f289d9d8798 in uuid_is_null () from /lib64/libuuid.so.1 Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.166.el6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libacl-2.2.49-6.el6.x86_64 libaio-0.3.107-10.el6.x86_64 libattr-2.4.44-7.el6.x86_64 libcom_err-1.41.12-21.el6.x86_64 libgcc-4.4.7-16.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libuuid-2.17.2-12.18.el6.x86_64 openssl-1.0.1e-39.el6.x86_64 sqlite-3.6.20-1.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007f289d9d8798 in uuid_is_null () from /lib64/libuuid.so.1 #1 0x00007f288a4929d3 in gf_uuid_is_null (req=<value optimized out>, iov=<value optimized out>, count=<value optimized out>, myframe=0x7f289be07ec8) at ../../../../libglusterfs/src/compat-uuid.h:45 #2 quota_enforcer_lookup_cbk (req=<value optimized out>, iov=<value optimized out>, count=<value optimized out>, myframe=0x7f289be07ec8) at quota-enforcer-client.c:174 #3 0x00007f289e018445 in rpc_clnt_handle_reply (clnt=0x7f288c03f8d0, pollin=0x7f288c0d5050) at rpc-clnt.c:766 #4 0x00007f289e0198f2 in rpc_clnt_notify (trans=<value optimized out>, mydata=0x7f288c03f900, event=<value optimized out>, data=<value optimized out>) at rpc-clnt.c:894 #5 0x00007f289e014ad8 in rpc_transport_notify (this=<value optimized out>, event=<value optimized out>, data=<value optimized out>) at rpc-transport.c:543 #6 0x00007f2892d7a255 in socket_event_poll_in (this=0x7f288c042a60) at socket.c:2290 #7 0x00007f2892d7be4d in socket_event_handler (fd=<value optimized out>, idx=<value optimized out>, data=0x7f288c042a60, poll_in=1, poll_out=0, poll_err=0) at socket.c:2403 #8 0x00007f289e2ad970 in event_dispatch_epoll_handler (data=0x7f289eed2d20) at event-epoll.c:575 #9 event_dispatch_epoll_worker (data=0x7f289eed2d20) at event-epoll.c:678 #10 0x00007f289d334a51 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f289cc9e96d in clone () from /lib64/libc.so.6 Hence moving back bug to assigned state
Changing state to modified as dependency bug# 1238049 is now fixed
Enable quota on volume testvol and set limit-usage to 1 GB of root of the volume [root@darkknightrises tmp]# gluster v quota vol0 list Path Hard-limit Soft-limit Used Available Soft-limit exceeded? Hard-limit exceeded? --------------------------------------------------------------------------------------------------------------------------- / 1.0GB 80% 0Bytes 1.0GB No Tried running command from one terminal dd if=/dev/zero of=f1 bs=1M count=500 oflag=sync On another terminal execute below rename operation while the write is still in progress for i in {1..50}; do ii=`expr $i + 1`; mv f$i f$ii; Quota list command display correct used and available size and Didn't see any crash too. [root@darkknight /]# gluster v quota testvol list Path Hard-limit Soft-limit Used Available Soft-limit exceeded? Hard-limit exceeded? --------------------------------------------------------------------------------------------------------------------------- / 1.0GB 80% 500.0MB 524.0MB No No Bug verified on build glusterfs-3.7.1-12.el7rhgs.x86_64. Marking this bug verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-1845.html