Bug 1236672

Summary: quota: brick crashes when create and remove performed in parallel
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Vijaikumar Mallikarjuna <vmallika>
Component: quotaAssignee: Vijaikumar Mallikarjuna <vmallika>
Status: CLOSED ERRATA QA Contact: Anil Shah <ashah>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rhgs-3.1CC: annair, asrivast, byarlaga, divya, mselvaga, rgowdapp, rhs-bugs, smohan, storage-qa-internal
Target Milestone: ---Keywords: ZStream
Target Release: RHGS 3.1.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.7.1-12 Doc Type: Bug Fix
Doc Text:
Previously, brick crashed when create, write, and remove operations were performed in parallel on a quota enabled volume. With this fix, brick does not crash even if create, write, and remove operations are performed parallel.
 Story Points: ---
Clone Of:
: 1244109 (view as bug list) Environment:
Last Closed: 2015-10-05 07:15:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1241807, 1244109, 1244116, 1251815    

Description Vijaikumar Mallikarjuna 2015-06-29 17:04:39 UTC 
With below test-case brick crashes in marker

Open 3 different terminal:

In first terminal create files/dirs

mkdir {1..1000}
touch 1/{1..1000}
mkdir 1/d{1..1000}

for i in {1..999}
do
    ii=`expr $i + 1`
    mv $i/* $ii/
done

In second terminal write to file
for i in {1..100}; do dd if=/dev/zero of=f$i bs=1M count=4 oflag=sync; done


In third terminal remove files
rm -rf /mnt/*



Here is the stack trace:
(gdb) bt
#0  list_del (this=0x7f7fe8019980, loc=0x7f7fa80d30d8, ctx=0x7f7fc808f800, 
    contri=0x7f7fc0007cf0, delta=<value optimized out>, 
    remove_xattr=<value optimized out>)
    at ../../../../libglusterfs/src/list.h:76
#1  mq_remove_contri (this=0x7f7fe8019980, loc=0x7f7fa80d30d8, 
    ctx=0x7f7fc808f800, contri=0x7f7fc0007cf0, delta=<value optimized out>, 
    remove_xattr=<value optimized out>) at marker-quota.c:2661
#2  0x00007f7fe719804c in mq_reduce_parent_size_task (opaque=0x7f7fa80d30d0)
    at marker-quota.c:3196
#3  0x00007f7ff759f8a2 in synctask_wrap (old_task=<value optimized out>)
    at syncop.c:376
#4  0x0000003bc7243bf0 in ?? () from /lib64/libc.so.6
#5  0x0000000000000000 in ?? ()


No quota errors in log
Comment 3 senaik 2015-07-06 06:40:14 UTC 
Version : glusterfs-3.7.1-7.el6rhs.x86_64

Faced brick crash while running dbench on quota enabled on the volumes and 256 snapshot creation was in progress : 

Core was generated by `/usr/sbin/glusterfsd -s rhs-arch-srv4.lab.eng.blr.redhat.com --volfile-id snapv'.
Program terminated with signal 11, Segmentation fault.
#0  list_del (this=0x7fd6ac017e80, loc=0x7fd610009a38, ctx=0x7fd6840091d0, contri=0x7fd684009fe0, delta=<value optimized out>, remove_xattr=<value optimized out>) at ../../../../libglusterfs/src/list.h:76
76		old->prev->next = old->next;
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.166.el6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libacl-2.2.49-6.el6.x86_64 libaio-0.3.107-10.el6.x86_64 libattr-2.4.44-7.el6.x86_64 libcom_err-1.41.12-21.el6.x86_64 libgcc-4.4.7-16.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libuuid-2.17.2-12.18.el6.x86_64 openssl-1.0.1e-39.el6.x86_64 sqlite-3.6.20-1.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  list_del (this=0x7fd6ac017e80, loc=0x7fd610009a38, ctx=0x7fd6840091d0, contri=0x7fd684009fe0, delta=<value optimized out>, remove_xattr=<value optimized out>) at ../../../../libglusterfs/src/list.h:76
#1  mq_remove_contri (this=0x7fd6ac017e80, loc=0x7fd610009a38, ctx=0x7fd6840091d0, contri=0x7fd684009fe0, delta=<value optimized out>, remove_xattr=<value optimized out>) at marker-quota.c:2634
#2  0x00007fd6aaf82a3c in mq_reduce_parent_size_task (opaque=0x7fd610009a30) at marker-quota.c:3135
#3  0x00007fd6bee271f2 in synctask_wrap (old_task=<value optimized out>) at syncop.c:381
#4  0x00007fd6bd78f8f0 in ?? () from /lib64/libc.so.6
#5  0x0000000000000000 in ?? ()
(gdb)
Comment 8 Anil Shah 2015-08-27 05:16:14 UTC 
Not seeing any bricks crash after continuous parallel create and rename operation.

Bug verified on build  glusterfs-3.7.1-12.el7rhgs.x86_64
Comment 10 errata-xmlrpc 2015-10-05 07:15:47 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1845.html