Description of problem: Samba crashed while running fs-sanity tests on gluster cifs mount. Version-Release number of selected component (if applicable): samba-4.1.17-1.el6rhs.x86_64 glusterfs-3.6.0.49-1.el6rhs.x86_64 How reproducible: Not sure. Trying to reproduce the issue Steps to Reproduce: Not sure of the exact steps. But running BVT test suite caused this issue. Actual results: Smbd crashed Expected results: Additional info: The crash back trace from /var/log/messages. I am also attaching the core file with this bug. ######################################## Mar 6 08:03:34 rhsauto019 smbd[7258]: INTERNAL ERROR: Signal 11 in pid 7258 (4.1.17) Mar 6 08:03:34 rhsauto019 smbd[7258]: Please read the Trouble-Shooting section of the Samba HOWTO Mar 6 08:03:34 rhsauto019 smbd[7258]: [2015/03/06 08:03:34.635190, 0] ../lib/util/fault.c:75(fault_report) Mar 6 08:03:34 rhsauto019 smbd[7258]: =============================================================== Mar 6 08:03:34 rhsauto019 smbd[7258]: [2015/03/06 08:03:34.635426, 0] ../source3/lib/util.c:785(smb_panic_s3) Mar 6 08:03:34 rhsauto019 smbd[7258]: PANIC (pid 7258): internal error Mar 6 08:03:34 rhsauto019 smbd[7258]: [2015/03/06 08:03:34.747698, 0] ../source3/lib/util.c:896(log_stack_trace) Mar 6 08:03:34 rhsauto019 smbd[7258]: BACKTRACE: 37 stack frames: Mar 6 08:03:34 rhsauto019 smbd[7258]: #0 /usr/lib64/libsmbconf.so.0(log_stack_trace+0x1a) [0x7f973292907a] Mar 6 08:03:34 rhsauto019 smbd[7258]: #1 /usr/lib64/libsmbconf.so.0(smb_panic_s3+0x23) [0x7f9732929143] Mar 6 08:03:34 rhsauto019 smbd[7258]: #2 /usr/lib64/libsamba-util.so.0(smb_panic+0x1a1) [0x7f97340a7ac1] Mar 6 08:03:34 rhsauto019 smbd[7258]: #3 /usr/lib64/libsamba-util.so.0(+0x18b82) [0x7f97340a7b82] Mar 6 08:03:34 rhsauto019 smbd[7258]: #4 /lib64/libpthread.so.0(+0xf710) [0x7f97342d4710] Mar 6 08:03:34 rhsauto019 smbd[7258]: #5 /usr/lib64/libglusterfs.so.0(inode_unref+0x16) [0x7f971da77826] Mar 6 08:03:34 rhsauto019 smbd[7258]: #6 /usr/lib64/glusterfs/3.6.0.49/xlator/cluster/replicate.so(afr_priv_destroy+0x19) [0x7f9718472059] Mar 6 08:03:34 rhsauto019 smbd[7258]: #7 /usr/lib64/glusterfs/3.6.0.49/xlator/cluster/replicate.so(fini+0x1e) [0x7f97184721ce] Mar 6 08:03:34 rhsauto019 smbd[7258]: #8 /usr/lib64/libglusterfs.so.0(+0x1ec41) [0x7f971da55c41] Mar 6 08:03:34 rhsauto019 smbd[7258]: #9 /usr/lib64/libglusterfs.so.0(+0x1ebd5) [0x7f971da55bd5] Mar 6 08:03:34 rhsauto019 smbd[7258]: #10 /usr/lib64/libglusterfs.so.0(+0x1ebd5) [0x7f971da55bd5] Mar 6 08:03:34 rhsauto019 smbd[7258]: #11 /usr/lib64/libglusterfs.so.0(+0x1ebd5) [0x7f971da55bd5] Mar 6 08:03:34 rhsauto019 smbd[7258]: #12 /usr/lib64/libglusterfs.so.0(+0x1ebd5) [0x7f971da55bd5] Mar 6 08:03:34 rhsauto019 smbd[7258]: #13 /usr/lib64/libglusterfs.so.0(+0x1ebd5) [0x7f971da55bd5] Mar 6 08:03:34 rhsauto019 smbd[7258]: #14 /usr/lib64/libglusterfs.so.0(+0x1ebd5) [0x7f971da55bd5] Mar 6 08:03:34 rhsauto019 smbd[7258]: #15 /usr/lib64/libglusterfs.so.0(+0x1ebd5) [0x7f971da55bd5] Mar 6 08:03:34 rhsauto019 smbd[7258]: #16 /usr/lib64/libglusterfs.so.0(+0x1ebd5) [0x7f971da55bd5] Mar 6 08:03:34 rhsauto019 smbd[7258]: #17 /usr/lib64/libglusterfs.so.0(glusterfs_graph_deactivate+0x15) [0x7f971da92d15] Mar 6 08:03:34 rhsauto019 smbd[7258]: #18 /usr/lib64/libgfapi.so.0(glfs_fini+0x36b) [0x7f971e10f50b] Mar 6 08:03:34 rhsauto019 smbd[7258]: #19 /usr/lib64/samba/vfs/glusterfs.so(+0x41f2) [0x7f971e3271f2] Mar 6 08:03:34 rhsauto019 smbd[7258]: #20 /usr/lib64/samba/libsmbd_base.so(close_cnum+0x75) [0x7f9733c9ac65] Mar 6 08:03:34 rhsauto019 smbd[7258]: #21 /usr/lib64/samba/libsmbd_base.so(smbXsrv_tcon_disconnect+0x14c) [0x7f9733cc808c] Mar 6 08:03:34 rhsauto019 smbd[7258]: #22 /usr/lib64/samba/libsmbd_base.so(reply_tdis+0x5f) [0x7f9733c4e66f] Mar 6 08:03:34 rhsauto019 smbd[7258]: #23 /usr/lib64/samba/libsmbd_base.so(+0x1347e4) [0x7f9733c987e4] Mar 6 08:03:34 rhsauto019 smbd[7258]: #24 /usr/lib64/samba/libsmbd_base.so(+0x1355d4) [0x7f9733c995d4] Mar 6 08:03:34 rhsauto019 smbd[7258]: #25 /usr/lib64/samba/libsmbd_base.so(+0x1364a1) [0x7f9733c9a4a1] Mar 6 08:03:34 rhsauto019 smbd[7258]: #26 /usr/lib64/libsmbconf.so.0(run_events_poll+0x2b7) [0x7f9732947d17] Mar 6 08:03:34 rhsauto019 smbd[7258]: #27 /usr/lib64/libsmbconf.so.0(+0x45166) [0x7f9732948166] Mar 6 08:03:34 rhsauto019 smbd[7258]: #28 /usr/lib64/libtevent.so.0(_tevent_loop_once+0x9d) [0x7f973133b49d] Mar 6 08:03:34 rhsauto019 smbd[7258]: #29 /usr/lib64/samba/libsmbd_base.so(smbd_process+0xce2) [0x7f9733c979a2] Mar 6 08:03:34 rhsauto019 smbd[7258]: #30 smbd(+0x88ac) [0x7f973470c8ac] Mar 6 08:03:34 rhsauto019 smbd[7258]: #31 /usr/lib64/libsmbconf.so.0(run_events_poll+0x2b7) [0x7f9732947d17] Mar 6 08:03:34 rhsauto019 smbd[7258]: #32 /usr/lib64/libsmbconf.so.0(+0x45166) [0x7f9732948166] Mar 6 08:03:34 rhsauto019 smbd[7258]: #33 /usr/lib64/libtevent.so.0(_tevent_loop_once+0x9d) [0x7f973133b49d] Mar 6 08:03:34 rhsauto019 smbd[7258]: #34 smbd(main+0x160b) [0x7f973470e4db] Mar 6 08:03:34 rhsauto019 smbd[7258]: #35 /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f9730fc2d5d] Mar 6 08:03:34 rhsauto019 smbd[7258]: #36 smbd(+0x5999) [0x7f9734709999] Mar 6 08:03:34 rhsauto019 smbd[7258]: [2015/03/06 08:03:34.756847, 0] ../source3/lib/dumpcore.c:317(dump_core) Mar 6 08:03:34 rhsauto019 smbd[7258]: dumping core in /var/log/samba/cores/smbd ################################ [root@rhsauto019 ~]# rpm -qa | grep samba samba-vfs-glusterfs-4.1.17-1.el6rhs.x86_64 samba-client-4.1.17-1.el6rhs.x86_64 samba-debuginfo-4.1.17-1.el6rhs.x86_64 samba-common-4.1.17-1.el6rhs.x86_64 samba-libs-4.1.17-1.el6rhs.x86_64 samba-4.1.17-1.el6rhs.x86_64 ############################################## [root@rhsauto019 ~]# rpm -qa | grep gluster gluster-nagios-addons-0.1.14-1.el6rhs.x86_64 glusterfs-server-3.6.0.49-1.el6rhs.x86_64 samba-vfs-glusterfs-4.1.17-1.el6rhs.x86_64 glusterfs-cli-3.6.0.49-1.el6rhs.x86_64 glusterfs-libs-3.6.0.49-1.el6rhs.x86_64 glusterfs-rdma-3.6.0.49-1.el6rhs.x86_64 rhs-tests-rhs-tests-beaker-rhs-gluster-qe-libs-2.3.12-0.noarch glusterfs-debuginfo-3.6.0.49-1.el6rhs.x86_64 glusterfs-3.6.0.49-1.el6rhs.x86_64 glusterfs-fuse-3.6.0.49-1.el6rhs.x86_64 glusterfs-geo-replication-3.6.0.49-1.el6rhs.x86_64 gluster-nagios-common-0.1.4-1.el6rhs.noarch vdsm-gluster-4.14.7.3-1.el6rhs.noarch glusterfs-devel-3.6.0.49-1.el6rhs.x86_64 glusterfs-api-3.6.0.49-1.el6rhs.x86_64
Here is the BT of the crash (gdb) bt #0 0x00007f9730fd6625 in raise () from /lib64/libc.so.6 #1 0x00007f9730fd7e05 in abort () from /lib64/libc.so.6 #2 0x00007f973293ef11 in dump_core () at ../source3/lib/dumpcore.c:336 #3 0x00007f9732929190 in smb_panic_s3 (why=<value optimized out>) at ../source3/lib/util.c:808 #4 0x00007f97340a7ac1 in smb_panic (why=0x7f97340b6c75 "internal error") at ../lib/util/fault.c:159 #5 0x00007f97340a7b82 in fault_report (sig=11) at ../lib/util/fault.c:77 #6 sig_fault (sig=11) at ../lib/util/fault.c:88 #7 <signal handler called> #8 inode_unref (inode=0x7f9708bd104c) at inode.c:451 #9 0x00007f9718472059 in afr_priv_destroy (priv=0x7f971002b6c0) at afr-common.c:4982 #10 0x00007f97184721ce in fini (this=<value optimized out>) at afr.c:484 #11 0x00007f971da55c41 in xlator_fini_rec (xl=0x7f9710008b10) at xlator.c:443 #12 0x00007f971da55bd5 in xlator_fini_rec (xl=0x7f9710009da0) at xlator.c:433 #13 0x00007f971da55bd5 in xlator_fini_rec (xl=0x7f971000b240) at xlator.c:433 #14 0x00007f971da55bd5 in xlator_fini_rec (xl=0x7f971000c6b0) at xlator.c:433 #15 0x00007f971da55bd5 in xlator_fini_rec (xl=0x7f971000d910) at xlator.c:433 #16 0x00007f971da55bd5 in xlator_fini_rec (xl=0x7f971000ebc0) at xlator.c:433 #17 0x00007f971da55bd5 in xlator_fini_rec (xl=0x7f9710010690) at xlator.c:433 #18 0x00007f971da55bd5 in xlator_fini_rec (xl=0x7f9710011980) at xlator.c:433 #19 0x00007f971da55bd5 in xlator_fini_rec (xl=0x7f9710012cb0) at xlator.c:433 #20 0x00007f971da92d15 in glusterfs_graph_deactivate (graph=<value optimized out>) at graph.c:340 #21 0x00007f971e10f50b in glfs_fini (fs=0x7f97355cb440) at glfs.c:943 #22 0x00007f971e3271f2 in glfs_clear_preopened (handle=<value optimized out>) at ../source3/modules/vfs_glusterfs.c:137 #23 vfs_gluster_disconnect (handle=<value optimized out>) at ../source3/modules/vfs_glusterfs.c:243 #24 0x00007f9733c9ac65 in close_cnum (conn=0x7f97355979a0, vuid=172) at ../source3/smbd/service.c:1133 #25 0x00007f9733cc808c in smbXsrv_tcon_disconnect (tcon=0x7f97355957c0, vuid=172) at ../source3/smbd/smbXsrv_tcon.c:977 #26 0x00007f9733c4e66f in reply_tdis (req=0x7f973628e920) at ../source3/smbd/reply.c:5396 #27 0x00007f9733c987e4 in switch_message (type=113 'q', req=0x7f973628e920) at ../source3/smbd/process.c:1564 #28 0x00007f9733c995d4 in construct_reply (sconn=0x7f9735587510, inbuf=<value optimized out>, nread=39, unread_bytes=0, seqnum=0, encrypted=false, deferred_pcd=0x0) at ../source3/smbd/process.c:1600 #29 process_smb (sconn=0x7f9735587510, inbuf=<value optimized out>, nread=39, unread_bytes=0, seqnum=0, encrypted=false, deferred_pcd=0x0) at ../source3/smbd/process.c:1851 #30 0x00007f9733c9a4a1 in smbd_server_connection_read_handler (sconn=0x7f9735587510, fd=32) at ../source3/smbd/process.c:2440 #31 0x00007f9732947d17 in run_events_poll (ev=0x7f97355782b0, pollrtn=<value optimized out>, pfds=0x7f9735575e80, num_pfds=3) at ../source3/lib/events.c:257 #32 0x00007f9732948166 in s3_event_loop_once (ev=0x7f97355782b0, location=<value optimized out>) at ../source3/lib/events.c:326 #33 0x00007f973133b49d in _tevent_loop_once () from /usr/lib64/libtevent.so.0 #34 0x00007f9733c979a2 in smbd_process (ev_ctx=0x7f97355782b0, msg_ctx=<value optimized out>, sock_fd=<value optimized out>, interactive=false) at ../source3/smbd/process.c:3695 #35 0x00007f973470c8ac in smbd_accept_connection (ev=0x7f97355782b0, fde=<value optimized out>, flags=<value optimized out>, private_data=<value optimized out>) at ../source3/smbd/server.c:610 #36 0x00007f9732947d17 in run_events_poll (ev=0x7f97355782b0, pollrtn=<value optimized out>, pfds=0x7f9735575e80, num_pfds=3) at ../source3/lib/events.c:257 #37 0x00007f9732948166 in s3_event_loop_once (ev=0x7f97355782b0, location=<value optimized out>) at ../source3/lib/events.c:326 #38 0x00007f973133b49d in _tevent_loop_once () from /usr/lib64/libtevent.so.0 #39 0x00007f973470e4db in smbd_parent_loop (argc=<value optimized out>, argv=<value optimized out>) at ../source3/smbd/server.c:934 #40 main (argc=<value optimized out>, argv=<value optimized out>) at ../source3/smbd/server.c:1566
RCA: Upstream master has AFR V2, Downstream 3.0.4 has both AFR V1 and AFR V1. While backporting the glfs_fini patches to downstream, the changes were made to AFR V2, only and not V1. Another patch needs to be sent to backport the AFR changes to AFR V1 as well.
patch poster for review @ https://code.engineering.redhat.com/gerrit/#/c/43451/
Another patch for the same issue @https://code.engineering.redhat.com/gerrit/#/c/43464/
Ran BVT on glusterfs-3.6.0.50-1 couple of time and not seeing the crash. Hence marking this as verified.
As it is a regression no doc text required
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-0682.html