Bug 1019469

Summary: Losing network connection causes segfault in winbindd
Product: [Fedora] Fedora Reporter: Jan-Michael Brummer <jan.brummer>
Component: sambaAssignee: Guenther Deschner <gdeschner>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 20CC: abokovoy, asn, dwmw2, gdeschner, jan.brummer, jlayton, jonabbey, sbose, ssorce
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: samba-4.1.3-2.fc20 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1040052 (view as bug list) Environment:
Last Closed: 2013-12-24 03:39:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1040052    
Attachments:
Description Flags
Fix NULL-pointer debug message none

Description Jan-Michael Brummer 2013-10-15 19:31:49 UTC
Description of problem:
winbindd is communicating to the server through wifi. As soon as the wifi connection is dropped the winbind daemon is crashing.

Version-Release number of selected component (if applicable):
Version    : 4.1.0
Release    : 1.fc20


How reproducible:
Always.


Steps to Reproduce:
1. Connect with winbindd
2. Drop/Disable wifi connection so that no connection is established
3.

Actual results:
winbindd crashes, pam_winbind refuses interaction and users cannot login

Expected results:
winbind goes offline, users can login through cached credentials

Additional info:
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: [2013/10/15 19:56:09.323703,  0] ../lib/util/fault.c:72(fault_report)
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: ===============================================================
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: [2013/10/15 19:56:09.324034,  0] ../lib/util/fault.c:73(fault_report)
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: INTERNAL ERROR: Signal 11 in pid 1006 (4.1.0)
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: Please read the Trouble-Shooting section of the Samba HOWTO
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: [2013/10/15 19:56:09.324073,  0] ../lib/util/fault.c:75(fault_report)
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: ===============================================================
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: [2013/10/15 19:56:09.324100,  0] ../source3/lib/util.c:785(smb_panic_s3)
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: PANIC (pid 1006): internal error
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: [2013/10/15 19:56:09.324557,  0] ../source3/lib/util.c:896(log_stack_trace)
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: BACKTRACE: 15 stack frames:
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: #0 /lib64/libsmbconf.so.0(log_stack_trace+0x1a) [0x7f435f91bd5a]
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: #1 /lib64/libsmbconf.so.0(smb_panic_s3+0x20) [0x7f435f91be30]
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: #2 /lib64/libsamba-util.so.0(smb_panic+0x2f) [0x7f4363c07fbf]
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: #3 /lib64/libsamba-util.so.0(+0x3c42c181d6) [0x7f4363c081d6]
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: #4 /lib64/libpthread.so.0(+0x350f00f750) [0x7f4364039750]
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: #5 /usr/sbin/winbindd(winbind_msg_offline+0x24c) [0x7f43644bd83c]
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: #6 /lib64/libsmbconf.so.0(messaging_dispatch_rec+0x5b) [0x7f435f922cbb]
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: #7 /lib64/libsmbconf.so.0(+0x3c4022c679) [0x7f435f923679]
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: #8 /lib64/libtevent.so.0(tevent_common_check_signal+0x247) [0x7f435db8af47]
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: #9 /lib64/libtevent.so.0(+0x354a808c2c) [0x7f435db8cc2c]
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: #10 /lib64/libtevent.so.0(+0x354a807157) [0x7f435db8b157]
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: #11 /lib64/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f435db87c1d]
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: #12 /usr/sbin/winbindd(main+0xa92) [0x7f436448e012]
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: #13 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f435d594d65]
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: #14 /usr/sbin/winbindd(+0x25705) [0x7f436448e705]
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: [2013/10/15 19:56:09.327331,  0] ../source3/lib/dumpcore.c:317(dump_core)
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: dumping core in /var/log/samba/cores/winbindd
Oct 15 19:56:09 jbrummer.company.com winbindd[1006]: Oct 15 19:56:09 jbrummer.company.com winbindd[2912]: [2013/10/15 19:56:09.331499,  0, pid=2912, effective(0, 0), real(0, 0), class=winbind] ../source3/winbindd/winbindd.c:234(winbindd_sig_term_handler)
Oct 15 19:56:09 jbrummer.company.com winbindd[1769]: [2013/10/15 19:56:09.331500,  0, pid=1769, effective(0, 0), real(0, 0), class=winbind] ../source3/winbindd/winbindd.c:234(winbindd_sig_term_handler)
Oct 15 19:56:09 jbrummer.company.com winbindd[2584]: [2013/10/15 19:56:09.331500,  0, pid=2584, effective(0, 0), real(0, 0), class=winbind] ../source3/winbindd/winbindd.c:234(winbindd_sig_term_handler)
Oct 15 19:56:09 jbrummer.company.com winbindd[1769]: Got sig[15] terminate (is_parent=0)
Oct 15 19:56:09 jbrummer.company.com winbindd[2584]: Got sig[15] terminate (is_parent=0)
Oct 15 19:56:09 jbrummer.company.com winbindd[2912]: Got sig[15] terminate (is_parent=0)
Oct 15 19:56:09 jbrummer.company.com systemd[1]: winbind.service: main process exited, code=dumped, status=6/ABRT

Comment 1 Andreas Schneider 2013-10-16 07:36:16 UTC
Could you please install debug packages and reproduce the error. In the log file you should fine the full backtrace AFTER the short one! Or you simply post the logfile which contains the crash!

Comment 2 Jan-Michael Brummer 2013-11-21 13:43:50 UTC
Attaching gdb to the running winbindd process gives the following backtrace information:

Program received signal SIGUSR1, User defined signal 1.
0x00007f4a74238f47 in kill () at ../sysdeps/unix/syscall-template.S:81
81	T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) bt full
#0  0x00007f4a74238f47 in kill () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1  0x00007f4a765b31fd in message_notify (procid=...) at ../source3/lib/messages_local.c:322
        pid = 1134
        ret = <optimized out>
        euid = <optimized out>
#2  messaging_tdb_send (msg_ctx=<optimized out>, pid=..., msg_type=<optimized out>, data=<optimized out>, backend=<optimized out>) at ../source3/lib/messages_local.c:426
        ctx = <optimized out>
        msg_array = 0x7f4a7c188760
        rec = <optimized out>
        status = <optimized out>
        key = {dptr = 0x7f4a7c191b20 "PID/1134", dsize = 9}
        tdb = 0x7f4a7c17ee90
        frame = 0x7f4a7c1902e0
        __FUNCTION__ = "messaging_tdb_send"
#3  0x00007f4a765b2a2c in messaging_send (msg_ctx=0x7f4a7c17c3d0, server=..., msg_type=msg_type@entry=1034, data=data@entry=0x7fffe4e5e920) at ../source3/lib/messages.c:365
No locals.
#4  0x00007f4a7b126c26 in winbindd_addr_changed (req=0x0) at ../source3/winbindd/winbindd.c:1285
        addrstr = "172.28.241.108\000\000\060\f\030|J\177\000\000VɁtJ\177\000\000\000\000\000\000\000\000\000\000\200ȁtJ"
        blob = {data = 0x7fffe4e5e950 "172.28.241.108", length = 15}
        state = 0x7f4a7c182ed0
        type = ADDRCHANGE_DEL
        addr = {ss_family = 2, __ss_align = 0, __ss_padding = '\000' <repeats 111 times>}
        status = <optimized out>
        __FUNCTION__ = "winbindd_addr_changed"
#5  0x00007f4a765bbd22 in addrchange_done (subreq=0x0) at ../source3/lib/addrchange.c:245
        req = 0x7f4a7c182380
        state = 0x7f4a7c183730
        addr = 0x7f4a7c1857c8
        h = 0x7f4a7c183740
        ifa = 0x7f4a7c183750
        rta = <optimized out>
        received = 68
        len = <optimized out>
        err = 0
        found = <optimized out>
        __FUNCTION__ = "addrchange_done"
#6  0x00007f4a7481ccab in epoll_event_loop_once () from /lib64/libtevent.so.0
No symbol table info available.
#7  0x00007f4a7481b157 in std_event_loop_once () from /lib64/libtevent.so.0
No symbol table info available.
#8  0x00007f4a74817c1d in _tevent_loop_once () from /lib64/libtevent.so.0
No symbol table info available.
#9  0x00007f4a7b11e012 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at ../source3/winbindd/winbindd.c:1582
        is_daemon = false
        Fork = true
        log_stdout = false
        no_process_group = false
        long_options = {{longName = 0x0, shortName = 0 '\000', argInfo = 4, arg = 0x7f4a74e39320 <poptHelpOptions>, val = 0, descrip = 0x7f4a7b1ac266 "Help options:", argDescrip = 0x0}, {longName = 0x7f4a7b1ac27b "stdout", shortName = 83 'S', argInfo = 0, arg = 0x0, val = 1003, 
            descrip = 0x7f4a7b1ac274 "Log to stdout", argDescrip = 0x0}, {longName = 0x7f4a7b1ac282 "foreground", shortName = 70 'F', argInfo = 0, arg = 0x0, val = 1001, descrip = 0x7f4a7b1ac28d "Daemon in foreground mode", argDescrip = 0x0}, {
            longName = 0x7f4a7b1ac2a7 "no-process-group", shortName = 0 '\000', argInfo = 0, arg = 0x0, val = 1002, descrip = 0x7f4a7b1ab690 "Don't create a new process group", argDescrip = 0x0}, {longName = 0x7f4a7b1f064d "daemon", shortName = 68 'D', argInfo = 0, arg = 0x0, 
            val = 1000, descrip = 0x7f4a7b1ac2b8 "Become a daemon (default)", argDescrip = 0x0}, {longName = 0x7f4a7b1ac2d2 "interactive", shortName = 105 'i', argInfo = 0, arg = 0x0, val = 105, descrip = 0x7f4a7b1ac2de "Interactive mode", argDescrip = 0x0}, {
            longName = 0x7f4a7b1ac2ef "no-caching", shortName = 110 'n', argInfo = 0, arg = 0x0, val = 110, descrip = 0x7f4a7b1ac2fa "Disable caching", argDescrip = 0x0}, {longName = 0x0, shortName = 0 '\000', argInfo = 4, arg = 0x7f4a7acb94a0 <popt_common_samba>, val = 0, 
            descrip = 0x7f4a7b1ac30a "Common samba options:", argDescrip = 0x0}, {longName = 0x0, shortName = 0 '\000', argInfo = 0, arg = 0x0, val = 0, descrip = 0x0, argDescrip = 0x0}}
        pc = 0x7f4a7aab3200 <DEBUGLEVEL_CLASS>
        opt = <optimized out>
        frame = 0x7f4a7c178170
        status = <optimized out>
        ok = <optimized out>
        __FUNCTION__ = "main"

Comment 3 Jan-Michael Brummer 2013-11-21 14:07:43 UTC
Created attachment 827210 [details]
Fix NULL-pointer debug message

Got the bug and wrote a patch for it. This specific bug only occures when using winbindd with debug log output. As you can see in the patch it was using the wrong structure to print out the data, a structure that is NULL in that case. Please add my fix.

Comment 4 Jonathan Abbey 2013-11-27 21:25:41 UTC
I believe I am seeing this same crash behavior in samba-winbind-4.1.2-1.fc20.x86_64.

I don't have debug logging enabled, but Samba is reacting to a deprecated syntax in my smb.conf file that appears to be causing it to go down the debug log path.

Here is the error stream that I was seeing before I fixed the deprecated syntax:

Nov 27 14:12:20 xportal3 winbindd[59389]: [2013/11/27 14:12:20.348385,  0] ../source3/passdb/pdb_interface.c:169(make_pdb_method_name)
Nov 27 14:12:20 xportal3 winbindd[59389]: No builtin nor plugin backend for tdbsam found
Nov 27 14:12:20 xportal3 winbindd[59389]: [2013/11/27 14:12:20.348699,  0] ../source3/lib/util.c:785(smb_panic_s3)
Nov 27 14:12:20 xportal3 winbindd[59389]: PANIC (pid 59389): pdb_get_methods: failed to get pdb methods for backend tdbsam
Nov 27 14:12:20 xportal3 winbindd[59389]:
Nov 27 14:12:20 xportal3 winbindd[59389]: [2013/11/27 14:12:20.349621,  0] ../source3/lib/util.c:896(log_stack_trace)
Nov 27 14:12:20 xportal3 winbindd[59389]: BACKTRACE: 32 stack frames:
Nov 27 14:12:20 xportal3 winbindd[59389]: #0 /lib64/libsmbconf.so.0(log_stack_trace+0x1a) [0x7fe4351b0d5a]
Nov 27 14:12:20 xportal3 winbindd[59389]: #1 /lib64/libsmbconf.so.0(smb_panic_s3+0x20) [0x7fe4351b0e30]
Nov 27 14:12:20 xportal3 winbindd[59389]: #2 /lib64/libsamba-util.so.0(smb_panic+0x2f) [0x7fe43949cfbf]
Nov 27 14:12:20 xportal3 winbindd[59389]: #3 /lib64/libpdb.so.0(+0x1fc8f) [0x7fe437812c8f]
Nov 27 14:12:20 xportal3 winbindd[59389]: #4 /lib64/libpdb.so.0(pdb_enum_alias_memberships+0x25) [0x7fe437813715]
Nov 27 14:12:20 xportal3 winbindd[59389]: #5 /usr/sbin/winbindd(_samr_GetAliasMembership+0xbc) [0x7fe439d8377c]
Nov 27 14:12:20 xportal3 winbindd[59389]: #6 /usr/sbin/winbindd(+0x94637) [0x7fe439d92637]
Nov 27 14:12:20 xportal3 winbindd[59389]: #7 /usr/sbin/winbindd(+0x77a86) [0x7fe439d75a86]
Nov 27 14:12:20 xportal3 winbindd[59389]: #8 /lib64/libdcerpc-binding.so.0(dcerpc_binding_handle_raw_call_send+0x8e) [0x7fe437a388ee]
Nov 27 14:12:20 xportal3 winbindd[59389]: #9 /lib64/libdcerpc-binding.so.0(dcerpc_binding_handle_call_send+0x22d) [0x7fe437a390ad]
Nov 27 14:12:20 xportal3 winbindd[59389]: #10 /lib64/libdcerpc-binding.so.0(dcerpc_binding_handle_call+0x6f) [0x7fe437a391ef]
Nov 27 14:12:20 xportal3 winbindd[59389]: #11 /usr/lib64/samba/libdcerpc-samba.so(dcerpc_samr_GetAliasMembership_r+0x1d) [0x7fe437ee661d]
Nov 27 14:12:20 xportal3 winbindd[59389]: #12 /usr/lib64/samba/libdcerpc-samba.so(dcerpc_samr_GetAliasMembership+0x1d) [0x7fe437ee681d]
Nov 27 14:12:20 xportal3 winbindd[59389]: #13 /usr/sbin/winbindd(rpc_lookup_useraliases+0xfd) [0x7fe439d4980d]
Nov 27 14:12:20 xportal3 winbindd[59389]: #14 /usr/sbin/winbindd(+0x51a7c) [0x7fe439d4fa7c]
Nov 27 14:12:20 xportal3 winbindd[59389]: #15 /usr/sbin/winbindd(+0x37d0d) [0x7fe439d35d0d]
Nov 27 14:12:20 xportal3 winbindd[59389]: #16 /usr/sbin/winbindd(_wbint_LookupUserAliases+0x48) [0x7fe439d54cd8]
Nov 27 14:12:20 xportal3 winbindd[59389]: #17 /usr/sbin/winbindd(+0x7b6e7) [0x7fe439d796e7]
Nov 27 14:12:20 xportal3 winbindd[59389]: #18 /usr/sbin/winbindd(winbindd_dual_ndrcmd+0xb1) [0x7fe439d544a1]
Nov 27 14:12:20 xportal3 winbindd[59389]: #19 /usr/sbin/winbindd(+0x532d4) [0x7fe439d512d4]
Nov 27 14:12:20 xportal3 winbindd[59389]: #20 /lib64/libtevent.so.0(+0x8cab) [0x7fe433421cab]
Nov 27 14:12:20 xportal3 winbindd[59389]: #21 /lib64/libtevent.so.0(+0x7157) [0x7fe433420157]
Nov 27 14:12:20 xportal3 winbindd[59389]: #22 /lib64/libtevent.so.0(_tevent_loop_once+0x8d) [0x7fe43341cc1d]
Nov 27 14:12:20 xportal3 winbindd[59389]: #23 /usr/sbin/winbindd(+0x5562a) [0x7fe439d5362a]
Nov 27 14:12:20 xportal3 winbindd[59389]: #24 /usr/sbin/winbindd(+0x55ce5) [0x7fe439d53ce5]
Nov 27 14:12:20 xportal3 winbindd[59389]: #25 /lib64/libtevent.so.0(tevent_common_loop_immediate+0xd4) [0x7fe43341d444]
Nov 27 14:12:20 xportal3 winbindd[59389]: #26 /lib64/libtevent.so.0(+0x8a67) [0x7fe433421a67]
Nov 27 14:12:20 xportal3 winbindd[59389]: #27 /lib64/libtevent.so.0(+0x7157) [0x7fe433420157]
Nov 27 14:12:20 xportal3 winbindd[59389]: #28 /lib64/libtevent.so.0(_tevent_loop_once+0x8d) [0x7fe43341cc1d]
Nov 27 14:12:20 xportal3 winbindd[59389]: #29 /usr/sbin/winbindd(main+0xa92) [0x7fe439d23012]
Nov 27 14:12:20 xportal3 winbindd[59389]: #30 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fe432e29d65]
Nov 27 14:12:20 xportal3 winbindd[59389]: #31 /usr/sbin/winbindd(+0x25705) [0x7fe439d23705]
Nov 27 14:12:20 xportal3 winbindd[59389]: [2013/11/27 14:12:20.360918,  0] ../source3/lib/dumpcore.c:317(dump_core)
Nov 27 14:12:20 xportal3 winbindd[59389]: dumping core in /var/log/samba/cores/winbindd
Nov 27 14:12:20 xportal3 winbindd[59389]:

Comment 5 Jonathan Abbey 2013-11-27 21:27:25 UTC
And the syntax in my smb.conf that had been causing the initial error was a line
that said

   idmap backend = ad

instead of

   idmap config * : backend = ad

The above version worked in Fedora 19, but triggered an error (and thus a crash) in Fedora 20.

Comment 6 Jonathan Abbey 2013-11-27 21:28:32 UTC
See also

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=383592

for some discussion on how upstream Samba has tightened up its syntax checking quite a bit, which interacts badly with this bug in particular.

Comment 7 Jonathan Abbey 2013-11-27 21:30:11 UTC
Scratch the last comment, I hadn't realized that the debian discussion was so old.  It is true that the idmap backend syntax seems to have gotten tighter between 19 and 20, though.

Comment 8 Alexander Bokovoy 2013-11-27 21:34:32 UTC
Thanks for the line of thought, I'll check this to see if there is indeed some error path caused by param refactoring.

Comment 9 Guenther Deschner 2013-12-09 14:37:32 UTC
Very good catch! Thanks for the patch, I'm pushing it to upstream samba right now.

Comment 10 Fedora Update System 2013-12-10 17:54:58 UTC
samba-4.1.3-2.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/samba-4.1.3-2.fc20

Comment 11 Fedora Update System 2013-12-11 16:46:09 UTC
Package samba-4.1.3-2.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing samba-4.1.3-2.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-23177/samba-4.1.3-2.fc20
then log in and leave karma (feedback).

Comment 12 Fedora Update System 2013-12-24 03:39:14 UTC
samba-4.1.3-2.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.