Description of problem: When a gluster volume quota command issue we got the following errors: 2014-07-18 21:12:59.706929] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-sas02-client-2: remote operation failed: No such file or directory. Path: <gfid:abd01245-73e1-4ef6-aba6-dc087cf0bccd> (abd01245-73e1-4ef6-aba6-dc087cf0bccd) After we disabled and re-enabled quota on a volume, bricks on that volume keep crashing and dumping core [2014-07-22 14:35:49.270983] W [marker-quota.c:1270:mq_get_parent_inode_local] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/features/locks.so(pl_common_inodelk+0x2af) [0x7f26c6abb4ef] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/performance/io-threads.so(iot_inodelk_cbk+0xb9) [0x7f26c6895ac9] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/features/marker.so(mq_inodelk_cbk+0xc5) [0x7f26c6479ef5]))) 0-sas02-marker: failed to build parent loc of <gfid:ac3de7cf-2bfa-4ed7-95e7-7d0e1dc151d1>/home [2014-07-22 14:35:49.281775] W [marker-quota.c:1270:mq_get_parent_inode_local] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/features/locks.so(pl_common_inodelk+0x2af) [0x7f26c6abb4ef] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/performance/io-threads.so(iot_inodelk_cbk+0xb9) [0x7f26c6895ac9] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/features/marker.so(mq_inodelk_cbk+0xc5) [0x7f26c6479ef5]))) 0-sas02-marker: failed to build parent loc of <gfid:ac3de7cf-2bfa-4ed7-95e7-7d0e1dc151d1>/home [2014-07-22 14:35:49.406603] W [marker-quota.c:1270:mq_get_parent_inode_local] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/features/locks.so(pl_common_inodelk+0x2af) [0x7f26c6abb4ef] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/performance/io-threads.so(iot_inodelk_cbk+0xb9) [0x7f26c6895ac9] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/features/marker.so(mq_inodelk_cbk+0xc5) [0x7f26c6479ef5]))) 0-sas02-marker: failed to build parent loc of <gfid:ac3de7cf-2bfa-4ed7-95e7-7d0e1dc151d1>/home [2014-07-22 14:35:49.410136] W [marker-quota.c:1270:mq_get_parent_inode_local] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/features/locks.so(pl_common_inodelk+0x2af) [0x7f26c6abb4ef] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/performance/io-threads.so(iot_inodelk_cbk+0xb9) [0x7f26c6895ac9] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/features/marker.so(mq_inodelk_cbk+0xc5) [0x7f26c6479ef5]))) 0-sas02-marker: failed to build parent loc of <gfid:ac3de7cf-2bfa-4ed7-95e7-7d0e1dc151d1>/home pending frames: frame : type(0) op(40) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(27) frame : type(0) op(40) frame : type(0) op(17) frame : type(0) op(17) frame : type(0) op(33) frame : type(0) op(40) frame : type(0) op(27) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2014-07-22 14:36:19configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.5.1 /lib/x86_64-linux-gnu/libc.so.6(+0x364a0)[0x7f26cb95b4a0] /lib/x86_64-linux-gnu/libc.so.6(+0x163321)[0x7f26cba88321] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/features/marker.so(mq_loc_fill_from_name+0x89)[0x7f26c6475c59] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/features/marker.so(mq_readdir_cbk+0x21f)[0x7f26c647673f] /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_readdir_cbk+0xc2)[0x7f26cc34d072] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/performance/io-threads.so(iot_readdir_cbk+0xc2)[0x7f26c6895c02] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/features/access-control.so(posix_acl_readdir_cbk+0xc2)[0x7f26c6cc6ca2] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/storage/posix.so(posix_do_readdir+0x1b8)[0x7f26c72fd8c8] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/storage/posix.so(posix_readdir+0x13)[0x7f26c72fdd43] /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_readdir+0x88)[0x7f26cc356a38] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/features/access-control.so(posix_acl_readdir+0x23c)[0x7f26c6cc908c] /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_readdir+0x88)[0x7f26cc356a38] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/performance/io-threads.so(iot_readdir_wrapper+0x150)[0x7f26c6899a80] /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(call_resume+0x1c5)[0x7f26cc36bcd5] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/performance/io-threads.so(iot_worker+0x146)[0x7f26c689da66] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f26cbcece9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f26cba193fd] --------- We restarted the bricks with "gluster volume start $vol force" Bricks restarted but then still keep crashing once a while. We also noticed some directories quota ran up and filled up over 100% Right now we have to keep watching the system as bricks will die anytime and we just have to keep restarting the volume.... Core file is 145MB and not able to upload.... Version-Release number of selected component (if applicable): How reproducible: N/A Steps to Reproduce: 1.gluster volume start $vol force 2.brick crashed 3.gluster volume start $vol force Actual results: Expected results: Additional info:
gdb --core=core 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 [2014-07-22 07:36:33.901295] W [marker-quota.c:1404:mq_release_parent_lock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/performance/io-threads.so(iot_lookup_cbk+0xd9) [0x7f8890c7f609] (-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_lookup_cbk+0xd9) [0x7f8896736f39] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/features/marker.so(mq_update_inode_contribution+0x44f) [0x7f889086080f]))) 0-sas02-marker: An operation during quota updation of path (/HyperionBackupTempSata02/hyperionprod005.shopzilla.laxhq/home/hyperion/Oracle/Middleware/EPMSystem11R1/common/JRE/Sun/1.6.0/lib/zi/Africa) failed (Invalid argument) [2014-07-22 07:37:01.342914] W [quota.c:3669:quota_statfs_validate_cbk] 0-sas02-quota: quota context is not present in inode (gfid:00000000-0000-0000-0000-000000000001) pending frames: frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(27) frame : type(0) op(27) frame : type(0) op(20) frame : type(0) op(27) frame : type(0) op(20) frame : type(0) op(40) frame : type(0) op(17) frame : type(0) op(17) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(31) frame : type(0) op(29) frame : type(0) op(29) frame : type(0) op(29) frame : type(0) op(30) frame : type(0) op(29) frame : type(0) op(29) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2014-07-22 07:37:09configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.5.1 /lib/x86_64-linux-gnu/libc.so.6(+0x364a0)[0x7f8895d424a0] /lib/x86_64-linux-gnu/libc.so.6(+0x163321)[0x7f8895e6f321] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/features/marker.so(mq_loc_fill_from_name+0x89)[0x7f889085cc59] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/features/marker.so(mq_readdir_cbk+0x21f)[0x7f889085d73f] /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_readdir_cbk+0xc2)[0x7f8896734072] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/performance/io-threads.so(iot_readdir_cbk+0xc2)[0x7f8890c7cc02] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/features/access-control.so(posix_acl_readdir_cbk+0xc2)[0x7f88910adca2] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/storage/posix.so(posix_do_readdir+0x1b8)[0x7f88916e48c8] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/storage/posix.so(posix_readdir+0x13)[0x7f88916e4d43] /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_readdir+0x88)[0x7f889673da38] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.1/xlator/features/access-control.so(posix_acl_readdir+0x23c)[0x7f88910b008c] /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_readdir+0x88)[0x7f889673da38]
volume status http://pastie.org/9412255 gdb --core=core http://pastie.org/9411618 brick log http://pastie.org/9403564 http://pastie.org/9412323
Gluster seems become unusable as whenever a write started happened, a brick crash. http://pastie.org/9413749 and also noticed these in glustershd.log http://pastie.org/9413757
The same crash was fixed as part of https://bugzilla.redhat.com/show_bug.cgi?id=1144315 in 3.5 and the fix will be available in glusterfs-3.5.3. Hence, moving the state of the bug to MODIFIED.
This bug is getting closed because the 3.5 is marked End-Of-Life. There will be no further updates to this version. Please open a new bug against a version that still receives bugfixes if you are still facing this issue in a more current release.