Bug 1000545
Summary: | Add brick operation is causing one of the smbd process in server to crash | |||
---|---|---|---|---|
Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Lalatendu Mohanty <lmohanty> | |
Component: | samba | Assignee: | Raghavendra Talur <rtalur> | |
Status: | CLOSED ERRATA | QA Contact: | Lalatendu Mohanty <lmohanty> | |
Severity: | urgent | Docs Contact: | ||
Priority: | high | |||
Version: | 2.1 | CC: | amarts, crh, sbhaloth, sdharane, shaines, spalai | |
Target Milestone: | --- | |||
Target Release: | --- | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | glusterfs-3.4.0.29rhs-1 | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 1002577 (view as bug list) | Environment: | ||
Last Closed: | 2013-09-23 22:32:16 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 1002577 |
Description
Lalatendu Mohanty
2013-08-23 15:47:43 UTC
We again checked and found that core files are getting generated, just after Add brick operation. Below log messages are copied from /var/log/messages for the core files getting generated. Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: From: http://www.samba.org/samba/docs/Samba3-HOWTO.pdf Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: [2013/08/27 14:40:15.207660, 0] lib/fault.c:51(fault_report) Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: =============================================================== Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: [2013/08/27 14:40:15.208137, 0] lib/util.c:1117(smb_panic) Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: PANIC (pid 8629): internal error Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: [2013/08/27 14:40:15.212885, 0] lib/util.c:1221(log_stack_trace) Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: BACKTRACE: 26 stack frames: Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #0 smbd(log_stack_trace+0x1a) [0x7f622b7de58a] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #1 smbd(smb_panic+0x2b) [0x7f622b7de65b] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #2 smbd(+0x41a0e4) [0x7f622b7cf0e4] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #3 /lib64/libc.so.6(+0x39dda32920) [0x7f6227688920] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #4 /usr/lib64/libgfapi.so.0(glfs_refresh_inode_safe+0x59) [0x7f6228efc609] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #5 /usr/lib64/libgfapi.so.0(glfs_migrate_fd_safe+0x80) [0x7f6228efc7c0] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #6 /usr/lib64/libgfapi.so.0(__glfs_migrate_fd+0x46) [0x7f6228efcc06] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #7 /usr/lib64/libgfapi.so.0(__glfs_resolve_fd+0x58) [0x7f6228efcd78] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #8 /usr/lib64/libgfapi.so.0(glfs_resolve_fd+0x73) [0x7f6228efd3e3] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #9 /usr/lib64/libgfapi.so.0(glfs_close+0x62) [0x7f6228efb4e2] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #10 smbd(fd_close+0x43) [0x7f622b50f383] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #11 smbd(+0x15f1f2) [0x7f622b5141f2] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #12 smbd(close_file+0x44d) [0x7f622b51579d] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #13 smbd(reply_close+0x8b) [0x7f622b4e0a8b] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #14 smbd(+0x178414) [0x7f622b52d414] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #15 smbd(+0x17882b) [0x7f622b52d82b] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #16 smbd(+0x178c45) [0x7f622b52dc45] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #17 smbd(run_events_poll+0x377) [0x7f622b7ed907] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #18 smbd(smbd_process+0x86d) [0x7f622b52b78d] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #19 smbd(+0x69502f) [0x7f622ba4a02f] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #20 smbd(run_events_poll+0x377) [0x7f622b7ed907] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #21 smbd(+0x438dbf) [0x7f622b7eddbf] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #22 /usr/lib64/libtevent.so.0(_tevent_loop_once+0x9d) [0x7f622802149d] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #23 smbd(main+0xf3b) [0x7f622ba4b32b] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #24 /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f6227674cdd] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: #25 smbd(+0xf4b99) [0x7f622b4a9b99] Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: [2013/08/27 14:40:15.215018, 0] lib/fault.c:372(dump_core) Aug 27 14:40:15 bvt-rhs1 GlusterFS[8629]: dumping core in /var/log/core Renaming the bug summary as "Add brick operation is causing smbd precess to crash" also below are more information to my last comment. Below are the operations can be done to reproduce the issue Create the volume 1. gluster volume create test-vol-3 replica 2 10.70.37.93:/rhs/brick1/test-vol-3-b1 10.70.37.136:/rhs/brick1/testvol-3-b1 2. gluster v start test-vol-3 Below few steps i.e. 3 to 6 are for AD integration 3.mount -t glusterfs -o acl 10.70.37.93:/test-vol-3 /mnt/testvol/ 4. mkdir /mnt/testvol/rhs-smb-test-vol-3 5. chgrp "domain users" /mnt/testvol/rhs-smb-test-vol-3 6. chmod 770 /mnt/testvol/rhs-smb-test-vol-3 Set "stat-prefetch off" as it is the recommended settings 7. gluster volume set test-vol-3 stat-prefetch off 8. Create files on the share using win7Client Add Brick 9. gluster v add-brick test-vol-3 10.70.37.131:/rhs/brick1/test-vol-3-b1 10.70.37.149:/rhs/brick1/test-vol-3-b1 step-9 , causes one of the smbd process to crash in the samba server and creates the core file. This in turn causes samba share not accessible from the win7client. Saw the same issue on Linux cifs client. To reproduce the issue : Create a dis-rep volume Do a cifs mount on linux client. Create files on mount point. Do add brick on the volume The core will get generated with add-brick operation. A work-around is available, so this is not likely a blocker. Note that much of the stack trace shown above is heavily invested in libgfapi. I don't know the internals of libgfapi, but based on the stack trace it would appear that this particular crash occurred when the vfs module called glfs_close(), which then needs to traverse a set of functions to resolve the Gluster file descriptor to "real" file descriptors in the underlying XFS file system(s). My guess is that file descriptor resolution is producing bad values. #4 /usr/lib64/libgfapi.so.0(glfs_refresh_inode_safe+0x59) [0x7f6228efc609] #5 /usr/lib64/libgfapi.so.0(glfs_migrate_fd_safe+0x80) [0x7f6228efc7c0] #6 /usr/lib64/libgfapi.so.0(__glfs_migrate_fd+0x46) [0x7f6228efcc06] #7 /usr/lib64/libgfapi.so.0(__glfs_resolve_fd+0x58) [0x7f6228efcd78] #8 /usr/lib64/libgfapi.so.0(glfs_resolve_fd+0x73) [0x7f6228efd3e3] #9 /usr/lib64/libgfapi.so.0(glfs_close+0x62) [0x7f6228efb4e2] #10 smbd(fd_close+0x43) [0x7f622b50f383] Samba (smbd) is trying to close an file handle, and the vfs module passes the request to glfs_close(), which needs to resolve the fd...but (at a guess) the resolution is getting messed up somehow. Note that the failure sequence is the same as given in Bug #1001614, which may be a duplicate. Patch posted for review at https://code.engineering.redhat.com/gerrit/#/c/12214/ I am not seeing this crash any-more with latest gluster packages i.e. glusterfs-server-3.4.0.30rhs-2.el6rhs.x86_64. Hence Marking this bug verified Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1262.html |