Bug 1045332

Summary:	SMB:smbd crash while updating glusterfs and samba packages.
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	surabhi <sbhaloth>
Component:	samba	Assignee:	Poornima G <pgurusid>
Status:	CLOSED DUPLICATE	QA Contact:	surabhi <sbhaloth>
Severity:	urgent	Docs Contact:
Priority:	high
Version:	2.1	CC:	grajaiya, lmohanty, vagarwal
Target Milestone:	---	Keywords:	Regression, ZStream
Target Release:	RHGS 2.1.2
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.4.0.53rhs	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-01-03 10:20:52 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description surabhi 2013-12-20 07:43:35 UTC

Description of problem:
While doing yum update from glusterfs version 51 to 52 and where samba package also gets updated to 167.5 observed smbd crash.
Update was finished and new glusterfs version was installed but core was generated by smbd.

Version-Release number of selected component (if applicable):
glusterfs-libs-3.4.0.52rhs-1.el6rhs.x86_64
glusterfs-server-3.4.0.52rhs-1.el6rhs.x86_64
glusterfs-api-3.4.0.52rhs-1.el6rhs.x86_64
samba-glusterfs-3.6.9-167.5.el6rhs.x86_64
glusterfs-devel-3.4.0.52rhs-1.el6rhs.x86_64
glusterfs-3.4.0.52rhs-1.el6rhs.x86_64
glusterfs-fuse-3.4.0.52rhs-1.el6rhs.x86_64
glusterfs-debuginfo-3.4.0.52rhs-1.el6rhs.x86_64
glusterfs-geo-replication-3.4.0.52rhs-1.el6rhs.x86_64
glusterfs-rdma-3.4.0.52rhs-1.el6rhs.x86_64
glusterfs-api-devel-3.4.0.52rhs-1.el6rhs.x86_64

How reproducible:
tried once

Steps to Reproduce:
1.stop all the volumes 
2.Do yum update
3.check the var/log/messages

Actual results:
Update happened but smbd generated core.
Dec 20 02:12:39 dhcp159-237 smbd[2461]:   INTERNAL ERROR: Signal 6 in pid 2461 (3.6.9-167.3.el6rhs)
Dec 20 02:12:39 dhcp159-237 smbd[2461]:   Please read the Trouble-Shooting section of the Samba3-HOWTO
Dec 20 02:12:39 dhcp159-237 smbd[2461]: [2013/12/20 02:12:39.724989,  0] lib/fault.c:50(fault_report)
Dec 20 02:12:39 dhcp159-237 smbd[23338]: [2013/12/20 02:12:39.725329,  0] smbd/close.c:1091(close_directory)
Dec 20 02:12:39 dhcp159-237 smbd[23338]:   Could not close dir! fname=rhsdata02, fd=-1, err=107=Transport endpoint is not connected
Dec 20 02:12:39 dhcp159-237 smbd[2461]: [2013/12/20 02:12:39.725479,  0] lib/fault.c:51(fault_report)
Dec 20 02:12:39 dhcp159-237 smbd[2461]:   ===============================================================
Dec 20 02:12:39 dhcp159-237 smbd[2461]:   ===============================================================#012Deregistering messaging pointer for type 784 - private_data=0x7f549bdc4600
Dec 20 02:12:39 dhcp159-237 smbd[2461]: [2013/12/20 02:12:39.725941,  0] lib/util.c:1117(smb_panic)
Dec 20 02:12:39 dhcp159-237 smbd[2461]:   PANIC (pid 2461): internal error
Dec 20 02:12:39 dhcp159-237 smbd[2461]:   PANIC (pid 2461): internal error#012Locking key 971A8A0200000000FC2D
Dec 20 02:12:39 dhcp159-237 smbd[3673]: [2013/12/20 02:12:39.728700,  0] smbd/close.c:1091(close_directory)
Dec 20 02:12:39 dhcp159-237 smbd[3673]:   Could not close dir! fname=., fd=-1, err=107=Transport endpoint is not connected
Dec 20 02:12:39 dhcp159-237 smbd[23753]: [2013/12/20 02:12:39.730997,  0] smbd/close.c:1091(close_directory)
Dec 20 02:12:39 dhcp159-237 smbd[23753]:   Could not close dir! fname=., fd=-1, err=107=Transport endpoint is not connected
Dec 20 02:12:39 dhcp159-237 smbd[2461]: [2013/12/20 02:12:39.735846,  0] lib/fault.c:372(dump_core)
Dec 20 02:12:39 dhcp159-237 smbd[2461]:   dumping core in /var/log/core
Dec 20 02:12:39 dhcp159-237 smbd[2461]:   dumping core in /var/log/core#012print_share_mode_table: share_mode_entry[6]: UNUSED pid = 3673, share_access = 0x7, private_options = 0x0, access_mask = 0x100081, mid = 0x2bb00a, type= 0x40, gen_id = 2325758141, uid = 11117, flags = 0, file_id 28a1a97:a3acfa0749322dfc:0, name_hash = 0x72f06930
Dec 20 02:12:39 dhcp159-237 smbd[2461]: [2013/12/20 02:12:39.737927,  0] smbd/close.c:1091(close_directory)
Dec 20 02:12:39 dhcp159-237 smbd[2461]:   Could not close dir! fname=rhsdata02, fd=-1, err=107=Transport endpoint is not connected
Dec 20 02:12:39 dhcp159-237 smbd[23753]: [2013/12/20 02:12:39.746813,  0] smbd/close.c:1091(close_directory)
Dec 20 02:12:39 dhcp159-237 smbd[23753]:   Could not close dir! fname=rhsdata02, fd=-1, err=107=Transport endpoint is not connected


Expected results:
update should happen and no crash should be seen

Additional info:

Comment 3 surabhi 2013-12-26 06:28:38 UTC

Saw the crash again:

Dec 24 05:17:15 dhcp159-237 smbd[23141]:   ===============================================================
Dec 24 05:17:15 dhcp159-237 smbd[23141]: [2013/12/24 05:17:15.805352,  0] lib/fault.c:48(fault_report)
Dec 24 05:17:15 dhcp159-237 smbd[23141]:   INTERNAL ERROR: Signal 6 in pid 23141 (3.6.9-167.8.el6rhs)
Dec 24 05:17:15 dhcp159-237 smbd[23141]:   Please read the Trouble-Shooting section of the Samba3-HOWTO
Dec 24 05:17:15 dhcp159-237 smbd[23141]: [2013/12/24 05:17:15.805896,  0] lib/fault.c:50(fault_report)
Dec 24 05:17:15 dhcp159-237 smbd[23141]:   
Dec 24 05:17:15 dhcp159-237 smbd[23141]:   From: http://www.samba.org/samba/docs/Samba3-HOWTO.pdf
Dec 24 05:17:15 dhcp159-237 smbd[23141]: [2013/12/24 05:17:15.806298,  0] lib/fault.c:51(fault_report)
Dec 24 05:17:15 dhcp159-237 smbd[23141]:   ===============================================================
Dec 24 05:17:15 dhcp159-237 smbd[23141]: [2013/12/24 05:17:15.806841,  0] lib/util.c:1117(smb_panic)
Dec 24 05:17:15 dhcp159-237 smbd[23141]:   PANIC (pid 23141): internal error
Dec 24 05:17:15 dhcp159-237 smbd[23141]: [2013/12/24 05:17:15.809559,  0] lib/util.c:1221(log_stack_trace)
Dec 24 05:17:15 dhcp159-237 smbd[23141]:   BACKTRACE: 11 stack frames:
Dec 24 05:17:15 dhcp159-237 smbd[23141]:    #0 smbd(log_stack_trace+0x1a) [0x7f3e9873f97a]
Dec 24 05:17:15 dhcp159-237 smbd[23141]:    #1 smbd(smb_panic+0x2b) [0x7f3e9873fa4b]
Dec 24 05:17:15 dhcp159-237 smbd[23141]:    #2 smbd(+0x41a4d4) [0x7f3e987304d4]
Dec 24 05:17:15 dhcp159-237 smbd[23141]:    #3 /lib64/libc.so.6(+0x352a232960) [0x7f3e945da960]
Dec 24 05:17:15 dhcp159-237 smbd[23141]:    #4 /lib64/libc.so.6(gsignal+0x35) [0x7f3e945da8e5]
Dec 24 05:17:15 dhcp159-237 smbd[23141]:    #5 /lib64/libc.so.6(abort+0x175) [0x7f3e945dc0c5]
Dec 24 05:17:15 dhcp159-237 smbd[23141]:    #6 /lib64/libc.so.6(+0x352a2707f7) [0x7f3e946187f7]
Dec 24 05:17:15 dhcp159-237 smbd[23141]:    #7 /lib64/libc.so.6(+0x352a276126) [0x7f3e9461e126]
Dec 24 05:17:15 dhcp159-237 smbd[23141]:    #8 /lib64/libc.so.6(+0x352a278c53) [0x7f3e94620c53]
Dec 24 05:17:15 dhcp159-237 smbd[23141]:    #9 /usr/lib64/libtalloc.so.2(+0x3531a02389) [0x7f3e9517f389]
Dec 24 05:17:15 dhcp159-237 smbd[23141]:    #10 /usr/lib64/libtalloc.so.2(+0x3531a02323) [0x7f3e9517f323]
Dec 24 05:17:15 dhcp159-237 smbd[23141]: [2013/12/24 05:17:15.811822,  0] lib/fault.c:372(dump_core)
Dec 24 05:17:15 dhcp159-237 smbd[23141]:   dumping core in /var/log/core
Dec 24 05:17:15 dhcp159-237 smbd[23141]: 
Dec 24 05:17:32 dhcp159-237 smbd[23263]: [2013/12/24 05:17:32.590817,  0] modules/vfs_glusterfs.c:282(vfs_gluster_connect)
Dec 24 05:17:32 dhcp159-237 smbd[23263]:   dis-rep: Initialized volume from server localhost
Dec 24 05:17:32 dhcp159-237 smbd[23264]: [2013/12/24 05:17:32.994028,  0] modules/vfs_glusterfs.c:282(vfs_gluster_connect)

Comment 4 surabhi 2014-01-03 09:57:24 UTC

Updated the glusterfs version from glusterfs-3.4.0.52rhs-1.el6rhs.x86_64 to 
glusterfs-3.4.0.53rhs-1.el6rhs.x86_64 and also updated the samba packages.Didn't see the crash.
Tried running I/O's for some time and ran some basic sanity test on smb mount and
also tried to update to the latest version from U1 build and didn't see any crash there as well so marking the bug verified in following version:
glusterfs-libs-3.4.0.53rhs-1.el6rhs.x86_64
glusterfs-geo-replication-3.4.0.53rhs-1.el6rhs.x86_64
glusterfs-devel-3.4.0.53rhs-1.el6rhs.x86_64
samba-glusterfs-3.6.9-167.9.el6rhs.x86_64
glusterfs-debuginfo-3.4.0.40rhs-1.el6rhs.x86_64
glusterfs-3.4.0.53rhs-1.el6rhs.x86_64
glusterfs-fuse-3.4.0.53rhs-1.el6rhs.x86_64
glusterfs-server-3.4.0.53rhs-1.el6rhs.x86_64
glusterfs-rdma-3.4.0.53rhs-1.el6rhs.x86_64
glusterfs-api-3.4.0.53rhs-1.el6rhs.x86_64
glusterfs-api-devel-3.4.0.53rhs-1.el6rhs.x86_64

Comment 5 Poornima G 2014-01-03 10:19:55 UTC

Patch merged downstream at https://code.engineering.redhat.com/gerrit/#/c/17893/

Comment 6 Poornima G 2014-01-03 10:20:52 UTC


*** This bug has been marked as a duplicate of bug 1046564 ***