Bug 1229740

Summary:	Brick process crashed and dumped core while running cthon tests
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Apeksha <akhakhar>
Component:	gluster-nfs	Assignee:	Pranith Kumar K <pkarampu>
Status:	CLOSED DUPLICATE	QA Contact:	Saurabh <saujain>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.1	CC:	mzywusko, ndevos, rhs-bugs, sasundar, skoduri, storage-qa-internal
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-06-10 12:02:01 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Apeksha 2015-06-09 14:23:57 UTC

Description of problem:
server crashed and dumped core while running cthon tests

backtrace of the core:
(gdb) bt
#0  0x00007f812c14f400 in pthread_spin_lock () from /lib64/libpthread.so.0
#1  0x00007f811f73f8ad in ?? () from /usr/lib64/glusterfs/3.7.1/xlator/storage/posix.so
#2  0x00007f812cdd0fa3 in dict_foreach_match () from /usr/lib64/libglusterfs.so.0
#3  0x00007f812cdd1038 in dict_foreach () from /usr/lib64/libglusterfs.so.0
#4  0x00007f811f73f335 in do_xattrop () from /usr/lib64/glusterfs/3.7.1/xlator/storage/posix.so
#5  0x00007f811f73f841 in posix_xattrop () from /usr/lib64/glusterfs/3.7.1/xlator/storage/posix.so
#6  0x00007f812cde10c3 in default_xattrop () from /usr/lib64/libglusterfs.so.0
#7  0x00007f812cde10c3 in default_xattrop () from /usr/lib64/libglusterfs.so.0
#8  0x00007f812cde10c3 in default_xattrop () from /usr/lib64/libglusterfs.so.0
#9  0x00007f812cde10c3 in default_xattrop () from /usr/lib64/libglusterfs.so.0
#10 0x00007f812cde10c3 in default_xattrop () from /usr/lib64/libglusterfs.so.0
#11 0x00007f812cde10c3 in default_xattrop () from /usr/lib64/libglusterfs.so.0
#12 0x00007f812cde10c3 in default_xattrop () from /usr/lib64/libglusterfs.so.0
#13 0x00007f812cde4782 in default_xattrop_resume () from /usr/lib64/libglusterfs.so.0
#14 0x00007f812cdffb50 in call_resume () from /usr/lib64/libglusterfs.so.0
#15 0x00007f811d9e3398 in iot_worker () from /usr/lib64/glusterfs/3.7.1/xlator/performance/io-threads.so
#16 0x00007f812c14aa51 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f812bab496d in clone () from /lib64/libc.so.6

Version-Release number of selected component (if applicable):
glusterfs-3.7.1-1.el6rhs.x86_64

How reproducible:
Always on this build


Steps to Reproduce:
1. Running automated cthon tests on glusterfs-3.7.1 build, server crashes and dumped core


Actual results: observed a core dump

Expected results:
No crashes and cthon tests should pass

Additional info:
Saw 2 cores on the same server, backtrace of both look same

bt of the 2nd core:
(gdb) bt
#0  0x00007f35c4206400 in pthread_spin_lock () from /lib64/libpthread.so.0
#1  0x00007f35b77f68ad in ?? () from /usr/lib64/glusterfs/3.7.1/xlator/storage/posix.so
#2  0x00007f35c4e87fa3 in dict_foreach_match () from /usr/lib64/libglusterfs.so.0
#3  0x00007f35c4e88038 in dict_foreach () from /usr/lib64/libglusterfs.so.0
#4  0x00007f35b77f6335 in do_xattrop () from /usr/lib64/glusterfs/3.7.1/xlator/storage/posix.so
#5  0x00007f35b77f6841 in posix_xattrop () from /usr/lib64/glusterfs/3.7.1/xlator/storage/posix.so
#6  0x00007f35c4e980c3 in default_xattrop () from /usr/lib64/libglusterfs.so.0
#7  0x00007f35c4e980c3 in default_xattrop () from /usr/lib64/libglusterfs.so.0
#8  0x00007f35c4e980c3 in default_xattrop () from /usr/lib64/libglusterfs.so.0
#9  0x00007f35c4e980c3 in default_xattrop () from /usr/lib64/libglusterfs.so.0
#10 0x00007f35c4e980c3 in default_xattrop () from /usr/lib64/libglusterfs.so.0
#11 0x00007f35c4e980c3 in default_xattrop () from /usr/lib64/libglusterfs.so.0
#12 0x00007f35c4e980c3 in default_xattrop () from /usr/lib64/libglusterfs.so.0
#13 0x00007f35c4e9b782 in default_xattrop_resume () from /usr/lib64/libglusterfs.so.0
#14 0x00007f35c4eb6b50 in call_resume () from /usr/lib64/libglusterfs.so.0
#15 0x00007f35b5a9a398 in iot_worker () from /usr/lib64/glusterfs/3.7.1/xlator/performance/io-threads.so
#16 0x00007f35c4201a51 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f35c3b6b96d in clone () from /lib64/libc.so.6

Comment 2 Apeksha 2015-06-09 14:45:24 UTC

sosreports and core : http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1229740/

Comment 3 Saurabh 2015-06-10 08:24:25 UTC

Well I tried to execute tests on my setup of 4 nodes and volume of type 6x2. The tests successfully finished. No crashes seen.

Comment 4 Soumya Koduri 2015-06-10 12:02:01 UTC


*** This bug has been marked as a duplicate of bug 1228496 ***