Bug 1235182 - quota: marker accounting miscalculated when renaming a file on with write is in progress
Summary: quota: marker accounting miscalculated when renaming a file on with write is ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: quota
Version: rhgs-3.1
Hardware: Unspecified
OS: Unspecified
medium
unspecified
Target Milestone: ---
: RHGS 3.1.1
Assignee: Vijaikumar Mallikarjuna
QA Contact: Anil Shah
URL:
Whiteboard:
Depends On:
Blocks: qe_tracker_everglades 1202842 1223636 1235195 1235990 1238049 1251815
TreeView+ depends on / blocked
 
Reported: 2015-06-24 08:31 UTC by Vijaikumar Mallikarjuna
Modified: 2016-09-17 12:36 UTC (History)
10 users (show)

Fixed In Version: glusterfs-3.7.1-12
Doc Type: Bug Fix
Doc Text:
When a rename operation is performed parallel with write operation, marker accounting goes wrong. It is fixed now, and quota shows proper size usage despite parallel operations.
Clone Of:
: 1235195 (view as bug list)
Environment:
Last Closed: 2015-10-05 07:14:40 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2015:1845 0 normal SHIPPED_LIVE Moderate: Red Hat Gluster Storage 3.1 update 2015-10-05 11:06:22 UTC

Description Vijaikumar Mallikarjuna 2015-06-24 08:31:49 UTC
With below test-case, marker accounting becomes bad:
1) Create a volume with 1 brick
2) fuse mount
3) on one terminal write some data
   dd if=/dev/zero of=f1 bs=1M count=500 oflag=sync
4) on another terminal execute below rename operation while the write is still in progress
    for i in {1..50}; do
        ii=`expr $i + 1`;
        mv f$i f$ii;
    done

Comment 8 Anil Shah 2015-07-04 13:50:09 UTC
Enable quota on volume vol0 and set limit-usage to 1 GB of root of the volume

[root@darkknightrises tmp]# gluster v quota vol0 list
                  Path                   Hard-limit Soft-limit   Used  Available  Soft-limit exceeded? Hard-limit exceeded?
---------------------------------------------------------------------------------------------------------------------------
/                                          1.0GB       80%      0Bytes   1.0GB              No         

Tried running command from one terminal 
   dd if=/dev/zero of=f1 bs=1M count=500 oflag=sync

On  another terminal execute below rename operation while the write is still in progress

    for i in {1..50}; do  ii=`expr $i + 1`; mv f$i f$ii;

brick process crash on one replica pair storage node.


bt from one storage node
=================================
(gdb) bt
#0  0x00007f5da09bb387 in _quota_enforcer_lookup (data=0x7f5dae24eca4) at quota-enforcer-client.c:259
#1  0x00007f5da09bb4ce in quota_enforcer_lookup (frame=0x7f5dae24eca4, this=<value optimized out>, 
    xdata=<value optimized out>, validate_cbk=<value optimized out>) at quota-enforcer-client.c:344
#2  0x00007f5da09af5cc in quota_validate (frame=0x7f5dae24eca4, inode=<value optimized out>, 
    this=0x7f5d9c019670, cbk_fn=0x7f5da09b9590 <quota_validate_cbk>) at quota.c:901
#3  0x00007f5da09af8ce in quota_check_size_limit (frame=0x7f5dae24eca4, ctx=0x7f5d74002660, 
    priv=0x7f5d9c037ea0, _inode=0x7f5d88e0906c, this=0x7f5d9c019670, op_errno=0x7f5d886843dc, 
    just_validated=0, delta=131072, local=0x7f5d9c038298, skip_check=0x7f5d886843d8) at quota.c:1080
#4  0x00007f5da09b72a2 in quota_check_limit (frame=0x7f5dae24eca4, inode=<value optimized out>, 
    this=0x7f5d9c019670, name=0x0, par=<value optimized out>) at quota.c:1204
#5  0x00007f5da09b9480 in quota_check_limit_continuation (parents=0x7f5d88684500, 
    inode=0x7f5d88e09108, op_ret=<value optimized out>, op_errno=0, data=0x7f5dae24eca4)
    at quota.c:959
#6  0x00007f5da09b29f3 in quota_build_ancestry_cbk (frame=0x7f5dae085760, 
    cookie=<value optimized out>, this=0x7f5d9c019670, op_ret=<value optimized out>, op_errno=0, 
    entries=<value optimized out>, xdata=0x0) at quota.c:742
#7  0x00007f5da0bd580a in marker_build_ancestry_cbk (frame=<value optimized out>, 
    cookie=<value optimized out>, this=<value optimized out>, op_ret=2, op_errno=0, 
    entries=0x7f5d886849f0, xdata=0x0) at marker.c:2791
#8  0x00007f5db06a0202 in default_readdirp_cbk (frame=0x7f5dae24ed50, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=2, op_errno=0, entries=<value optimized out>, xdata=0x0)
    at defaults.c:1237
#9  0x00007f5da1415860 in up_readdir_cbk (frame=0x7f5dae24eea8, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=2, op_errno=0, entries=0x7f5d886849f0, xdata=0x0)
    at upcall.c:1292
#10 0x00007f5da1629555 in pl_readdirp_cbk (frame=0x7f5dae24fb6c, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=2, op_errno=0, entries=0x7f5d886849f0, xdata=0x0)
    at posix.c:2202
#11 0x00007f5da1841e7d in posix_acl_readdirp_cbk (frame=0x7f5dae24e89c, cookie=<value optimized out>, 
    this=<value optimized out>, op_ret=2, op_errno=0, entries=0x7f5d886849f0, xdata=0x0)
    at posix-acl.c:1591
---Type <return> to continue, or q <return> to quit---
#12 0x00007f5da1a5396c in br_stub_readdirp_cbk (frame=0x7f5dae24d9d4, cookie=<value optimized out>, 
    this=0x7f5da1841c00, op_ret=2, op_errno=0, entries=0x7f5d886849f0, dict=0x0)
    at bit-rot-stub.c:2249
#13 0x00007f5da2f85c83 in posix_readdirp (frame=0x7f5dae24da80, this=<value optimized out>, 
    fd=<value optimized out>, size=<value optimized out>, off=<value optimized out>, 
    dict=<value optimized out>) at posix.c:5580
#14 0x00007f5db0699633 in default_readdirp (frame=0x7f5dae24da80, this=0x7f5d9c0092a0, 
    fd=0x7f5d9c0bcce4, size=0, off=<value optimized out>, xdata=<value optimized out>)
    at defaults.c:2101
#15 0x00007f5db0699633 in default_readdirp (frame=0x7f5dae24da80, this=0x7f5d9c00a860, 
    fd=0x7f5d9c0bcce4, size=0, off=<value optimized out>, xdata=<value optimized out>)
    at defaults.c:2101
#16 0x00007f5db0699633 in default_readdirp (frame=0x7f5dae24da80, this=0x7f5d9c00d0f0, 
    fd=0x7f5d9c0bcce4, size=0, off=<value optimized out>, xdata=<value optimized out>)
    at defaults.c:2101
#17 0x00007f5da1a4ba44 in br_stub_readdirp (frame=0x7f5dae24d9d4, this=0x7f5d9c00f000, 
    fd=0x7f5d9c0bcce4, size=0, offset=0, dict=0x7f5dadc4ccb4) at bit-rot-stub.c:2284
#18 0x00007f5da183e7ad in posix_acl_readdirp (frame=0x7f5dae24e89c, this=0x7f5d9c010490, 
    fd=0x7f5d9c0bcce4, size=0, offset=0, dict=<value optimized out>) at posix-acl.c:1625
#19 0x00007f5da162a755 in pl_readdirp (frame=0x7f5dae24fb6c, this=0x7f5d9c011820, fd=0x7f5d9c0bcce4, 
    size=0, offset=0, xdata=0x7f5dadc4ccb4) at posix.c:2213
#20 0x00007f5da1410ea7 in up_readdirp (frame=0x7f5dae24eea8, this=0x7f5d9c012ba0, fd=0x7f5d9c0bcce4, 
    size=0, off=0, dict=0x7f5dadc4ccb4) at upcall.c:1342
#21 0x00007f5db069bf43 in default_readdirp_resume (frame=0x7f5dae24ed50, this=0x7f5d9c014000, 
    fd=0x7f5d9c0bcce4, size=0, off=0, xdata=0x7f5dadc4ccb4) at defaults.c:1657
#22 0x00007f5db06bc580 in call_resume (stub=0x7f5dadcda60c) at call-stub.c:2576
#23 0x00007f5da1208541 in iot_worker (data=0x7f5d9c053f70) at io-threads.c:215
#24 0x00007f5daf77ca51 in ?? ()
#25 0x00007f5d88685700 in ?? ()
#26 0x0000000000000000 in ?? ()


=====================================================
bt from another storage node
======================================================

Program terminated with signal 11, Segmentation fault.
#0  0x00007f289d9d8798 in uuid_is_null () from /lib64/libuuid.so.1
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.166.el6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libacl-2.2.49-6.el6.x86_64 libaio-0.3.107-10.el6.x86_64 libattr-2.4.44-7.el6.x86_64 libcom_err-1.41.12-21.el6.x86_64 libgcc-4.4.7-16.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libuuid-2.17.2-12.18.el6.x86_64 openssl-1.0.1e-39.el6.x86_64 sqlite-3.6.20-1.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  0x00007f289d9d8798 in uuid_is_null () from /lib64/libuuid.so.1
#1  0x00007f288a4929d3 in gf_uuid_is_null (req=<value optimized out>, iov=<value optimized out>, 
    count=<value optimized out>, myframe=0x7f289be07ec8)
    at ../../../../libglusterfs/src/compat-uuid.h:45
#2  quota_enforcer_lookup_cbk (req=<value optimized out>, iov=<value optimized out>, 
    count=<value optimized out>, myframe=0x7f289be07ec8) at quota-enforcer-client.c:174
#3  0x00007f289e018445 in rpc_clnt_handle_reply (clnt=0x7f288c03f8d0, pollin=0x7f288c0d5050)
    at rpc-clnt.c:766
#4  0x00007f289e0198f2 in rpc_clnt_notify (trans=<value optimized out>, mydata=0x7f288c03f900, 
    event=<value optimized out>, data=<value optimized out>) at rpc-clnt.c:894
#5  0x00007f289e014ad8 in rpc_transport_notify (this=<value optimized out>, 
    event=<value optimized out>, data=<value optimized out>) at rpc-transport.c:543
#6  0x00007f2892d7a255 in socket_event_poll_in (this=0x7f288c042a60) at socket.c:2290
#7  0x00007f2892d7be4d in socket_event_handler (fd=<value optimized out>, idx=<value optimized out>, 
    data=0x7f288c042a60, poll_in=1, poll_out=0, poll_err=0) at socket.c:2403
#8  0x00007f289e2ad970 in event_dispatch_epoll_handler (data=0x7f289eed2d20) at event-epoll.c:575
#9  event_dispatch_epoll_worker (data=0x7f289eed2d20) at event-epoll.c:678
#10 0x00007f289d334a51 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f289cc9e96d in clone () from /lib64/libc.so.6


Hence moving back bug to assigned state

Comment 19 Vijaikumar Mallikarjuna 2015-08-14 06:22:39 UTC
Changing state to modified as dependency bug# 1238049 is now fixed

Comment 20 Anil Shah 2015-08-26 05:11:33 UTC
Enable quota on volume testvol and set limit-usage to 1 GB of root of the volume

[root@darkknightrises tmp]# gluster v quota vol0 list
                  Path                   Hard-limit Soft-limit   Used  Available  Soft-limit exceeded? Hard-limit exceeded?
---------------------------------------------------------------------------------------------------------------------------
/                                          1.0GB       80%      0Bytes   1.0GB              No         

Tried running command from one terminal 
   dd if=/dev/zero of=f1 bs=1M count=500 oflag=sync

On  another terminal execute below rename operation while the write is still in progress

    for i in {1..50}; do  ii=`expr $i + 1`; mv f$i f$ii;

Quota list command display correct used and available size and Didn't see any crash too.

[root@darkknight /]# gluster v quota testvol list
                  Path                   Hard-limit Soft-limit   Used  Available  Soft-limit exceeded? Hard-limit exceeded?
---------------------------------------------------------------------------------------------------------------------------
/                                          1.0GB       80%     500.0MB 524.0MB              No                   No


Bug verified on build glusterfs-3.7.1-12.el7rhgs.x86_64.
Marking this bug verified.

Comment 22 errata-xmlrpc 2015-10-05 07:14:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1845.html


Note You need to log in before you can comment on or make changes to this bug.