Created attachment 1230656 [details] Samba log file (for one client) Description of problem: After upgrading to "3.9.0-ubuntu1~xenial5" for GlusterFS and "2:4.3.11+dfsg-0ubuntu0.16.04.2glusterfs3.9.0xenial1" for the Samba VFS Module, I'm experiencing crashes of the smbd deamon. System was working fine with 3.7.x Version-Release number of selected component (if applicable): GlusterFS: 3.9.0-ubuntu1~xenial5 Samba: 2:4.3.11+dfsg-0ubuntu0.16.04.2glusterfs3.9.0xenial1 from Andre Bauer PPA ( http://ppa.launchpad.net/monotek/samba-glusterfs-3.9/ubuntu xenial/main amd64 Packages) How reproducible: Crash is immediate, as soon as the user access the Samba share exposed through VFS Steps to Reproduce: 1. Expose a samba share (stored in a GlusterFS volume) through Gluster VFS in smb.conf: path = /share kernel share modes = no vfs objects = glusterfs glusterfs:volfile_server = localhost glusterfs:volume = smbshare 2. Access the share from a samba or windows client 3. Actual results: smbd daemon crashes. Here is the dump of the samba panic action script. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 0x00007fe9d59f651b in __GI___waitpid (pid=3746, stat_loc=stat_loc@entry=0x7ffd24482ed0, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29 #0 0x00007fe9d59f651b in __GI___waitpid (pid=3746, stat_loc=stat_loc@entry=0x7ffd24482ed0, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29 #1 0x00007fe9d596ffbb in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:148 #2 0x00007fe9d841a841 in smb_panic_s3 () from /usr/lib/x86_64-linux-gnu/samba/libsmbregistry.so.0 #3 0x00007fe9d918df1f in smb_panic () from /usr/lib/x86_64-linux-gnu/libsamba-util.so.0 #4 0x00007fe9d918e136 in ?? () from /usr/lib/x86_64-linux-gnu/libsamba-util.so.0 #5 <signal handler called> #6 0x00007fe9d5960428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54 #7 0x00007fe9d596202a in __GI_abort () at abort.c:89 #8 0x00007fe9d59a27ea in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7fe9d5abb2e0 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175 #9 0x00007fe9d59aae0a in malloc_printerr (ar_ptr=<optimized out>, ptr=<optimized out>, str=0x7fe9d5ab80b2 "free(): invalid pointer", action=3) at malloc.c:5004 #10 _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3865 #11 0x00007fe9d59ae98c in __GI___libc_free (mem=<optimized out>) at malloc.c:2966 #12 0x00007fe9d8d5ff65 in ?? () from /usr/lib/x86_64-linux-gnu/samba/libsmbd-base.so.0 #13 0x00007fe9d8d60ec7 in ?? () from /usr/lib/x86_64-linux-gnu/samba/libsmbd-base.so.0 #14 0x00007fe9d8d61a11 in ?? () from /usr/lib/x86_64-linux-gnu/samba/libsmbd-base.so.0 #15 0x00007fe9d8d61f10 in make_connection () from /usr/lib/x86_64-linux-gnu/samba/libsmbd-base.so.0 #16 0x00007fe9d8d103f2 in reply_tcon_and_X () from /usr/lib/x86_64-linux-gnu/samba/libsmbd-base.so.0 #17 0x00007fe9d8d5b1d7 in ?? () from /usr/lib/x86_64-linux-gnu/samba/libsmbd-base.so.0 #18 0x00007fe9d8d5cf23 in ?? () from /usr/lib/x86_64-linux-gnu/samba/libsmbd-base.so.0 #19 0x00007fe9d8d5e58c in ?? () from /usr/lib/x86_64-linux-gnu/samba/libsmbd-base.so.0 #20 0x00007fe9d70ad917 in run_events_poll () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0 #21 0x00007fe9d70adb77 in ?? () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0 #22 0x00007fe9d5cf7d3d in _tevent_loop_once () from /usr/lib/x86_64-linux-gnu/libtevent.so.0 #23 0x00007fe9d5cf7edb in tevent_common_loop_wait () from /usr/lib/x86_64-linux-gnu/libtevent.so.0 #24 0x00007fe9d8d5f8e8 in smbd_process () from /usr/lib/x86_64-linux-gnu/samba/libsmbd-base.so.0 #25 0x000055b1383f8e12 in ?? () #26 0x00007fe9d70ad917 in run_events_poll () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0 #27 0x00007fe9d70adb77 in ?? () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0 #28 0x00007fe9d5cf7d3d in _tevent_loop_once () from /usr/lib/x86_64-linux-gnu/libtevent.so.0 #29 0x00007fe9d5cf7edb in tevent_common_loop_wait () from /usr/lib/x86_64-linux-gnu/libtevent.so.0 #30 0x000055b1383f7099 in main () Expected results: No crash of smbd Additional info: Shares are stored on a GlusterFS replicated volume and are exposed through Samba VFS module.
This bug is getting closed because GlusterFS-3.9 has reached its end-of-life [1]. Note: This bug is being closed using a script. No verification has been performed to check if it still exists on newer releases of GlusterFS. If this bug still exists in newer GlusterFS releases, please open a new bug against the newer release. [1]: https://www.gluster.org/community/release-schedule/
I have gotten this same error. I tracked it down to the pub_glfs_realpath() function in libgfapi being changed to use jemalloc instead of libc's malloc() in GlusterFS commit b07c5324161f2fda22d5903db61b9711cf949229. Because of that, any memory that is returned by this function can no longer be deallocated through free(). Your crash is because the older versions of Samba deallocate the memory with free(). In commit 92a0a56c3852726e0812d260e043957c879aefa4, the GlusterFS VFS module was changed to preallocate the memory for the result before calling the function, thus bypassing jemalloc. The easy fix is to use a newer version of Samba that contains this commit or a cherry-pick of it. That would be Samba 4.4.9+, 4.5.2+, or 4.6.0+.
Thanks for the info! So this should work with the ubuntu zesty packages again: https://launchpad.net/~monotek/+archive/ubuntu/samba-glusterfs-3.9
Thanks Jack for the info and André for the new kit. As always perfect service :-) But it means for me that I have to wait for the official release of Zesty to test it. I will post the results on this bug in the coming weeks.
So, I did the upgrade to Zesty last week and it's now working perfectly well. The current versions I use are: - glusterfs-client/zesty, 3.10.1-ubuntu1~zesty1 (from Gluster team's ppa) - glusterfs-server/zesty, 3.10.1-ubuntu1~zesty1 (from Gluster team's ppa) - samba-vfs-modules/zesty,2:4.5.4+dfsg-1ubuntu2glusterfs3.10.1zesty1 (from André's ppa). Thanks once again for your support.