Bug 1375973 - smbd crashes on startup with libtevent 0.9.30
Summary: smbd crashes on startup with libtevent 0.9.30
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: libtevent
Version: 25
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Jakub Hrozek
QA Contact: Fedora Extras Quality Assurance
URL: https://retrace.fedoraproject.org/faf...
Whiteboard: abrt_hash:417cbecd0f63150e4b4626afbe8...
: 1384337 1385327 1387517 (view as bug list)
Depends On:
Blocks: 1415574
TreeView+ depends on / blocked
 
Reported: 2016-09-14 11:41 UTC by Mikkel Lauritsen
Modified: 2017-06-27 14:45 UTC (History)
23 users (show)

Fixed In Version: samba-4.5.0-3.fc25
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1415574 (view as bug list)
Environment:
Last Closed: 2016-10-19 17:27:06 UTC
Type: ---
Embargoed:

Attachments (Terms of Use)
File: backtrace (26.34 KB, text/plain)
2016-09-14 11:42 UTC, Mikkel Lauritsen 		no flags Details
File: cgroup (256 bytes, text/plain)
2016-09-14 11:42 UTC, Mikkel Lauritsen 		no flags Details
File: core_backtrace (4.23 KB, text/plain)
2016-09-14 11:42 UTC, Mikkel Lauritsen 		no flags Details
File: dso_list (13.60 KB, text/plain)
2016-09-14 11:42 UTC, Mikkel Lauritsen 		no flags Details
File: environ (227 bytes, text/plain)
2016-09-14 11:42 UTC, Mikkel Lauritsen 		no flags Details
File: limits (1.29 KB, text/plain)
2016-09-14 11:42 UTC, Mikkel Lauritsen 		no flags Details
File: maps (55.52 KB, text/plain)
2016-09-14 11:42 UTC, Mikkel Lauritsen 		no flags Details
File: mountinfo (3.19 KB, text/plain)
2016-09-14 11:42 UTC, Mikkel Lauritsen 		no flags Details
File: namespaces (102 bytes, text/plain)
2016-09-14 11:42 UTC, Mikkel Lauritsen 		no flags Details
File: open_fds (1.35 KB, text/plain)
2016-09-14 11:42 UTC, Mikkel Lauritsen 		no flags Details
File: proc_pid_status (1.08 KB, text/plain)
2016-09-14 11:42 UTC, Mikkel Lauritsen 		no flags Details
File: var_log_messages (4.20 KB, text/plain)
2016-09-14 11:42 UTC, Mikkel Lauritsen 		no flags Details
New backtrace (26.66 KB, text/plain)
2016-09-19 12:16 UTC, Mikkel Lauritsen 		no flags Details
Valgrind log file (62.73 KB, text/plain)
2016-09-22 06:27 UTC, Mikkel Lauritsen 		no flags Details
Valgrind Log with Debug Info (94.65 KB, text/plain)
2016-09-22 16:12 UTC, J W 		no flags Details
Show Obsolete (1) View All


Links
System ID Private Priority Status Summary Last Updated
Samba Project 12283 0 None None None 2019-06-24 07:33:43 UTC

Description Mikkel Lauritsen 2016-09-14 11:41:52 UTC 
Description of problem:


Version-Release number of selected component:
samba-4.5.0-0.0.rc1.fc25

Additional info:
reporter:       libreport-2.8.0
backtrace_rating: 4
cmdline:        /usr/sbin/smbd
crash_function: dump_core
executable:     /usr/sbin/smbd
global_pid:     7295
kernel:         4.8.0-0.rc5.git1.1.fc25.x86_64
pkg_fingerprint: 4089 D8F2 FDB1 9C98
pkg_vendor:     Fedora Project
runlevel:       N 3
type:           CCpp
uid:            0

Truncated backtrace:
Thread no. 1 (10 frames)
 #2 dump_core at ../source3/lib/dumpcore.c:322
 #3 smb_panic_s3 at ../source3/lib/util.c:814
 #4 smb_panic at ../lib/util/fault.c:166
 #5 fault_report at ../lib/util/fault.c:83
 #6 sig_fault at ../lib/util/fault.c:94
 #8 tevent_debug at ../tevent_debug.c:89
 #9 tevent_common_loop_timer_delay at ../tevent_timed.c:330
 #10 run_events_poll at ../source3/lib/events.c:199
 #11 s3_event_loop_once at ../source3/lib/events.c:303
 #12 _tevent_loop_once at ../tevent.c:680

Potential duplicate: bug 1186771
Comment 1 Mikkel Lauritsen 2016-09-14 11:42:01 UTC 
Created attachment 1200792 [details]
File: backtrace
Comment 2 Mikkel Lauritsen 2016-09-14 11:42:03 UTC 
Created attachment 1200793 [details]
File: cgroup
Comment 3 Mikkel Lauritsen 2016-09-14 11:42:04 UTC 
Created attachment 1200794 [details]
File: core_backtrace
Comment 4 Mikkel Lauritsen 2016-09-14 11:42:06 UTC 
Created attachment 1200795 [details]
File: dso_list
Comment 5 Mikkel Lauritsen 2016-09-14 11:42:08 UTC 
Created attachment 1200796 [details]
File: environ
Comment 6 Mikkel Lauritsen 2016-09-14 11:42:09 UTC 
Created attachment 1200797 [details]
File: limits
Comment 7 Mikkel Lauritsen 2016-09-14 11:42:11 UTC 
Created attachment 1200798 [details]
File: maps
Comment 8 Mikkel Lauritsen 2016-09-14 11:42:12 UTC 
Created attachment 1200799 [details]
File: mountinfo
Comment 9 Mikkel Lauritsen 2016-09-14 11:42:14 UTC 
Created attachment 1200800 [details]
File: namespaces
Comment 10 Mikkel Lauritsen 2016-09-14 11:42:15 UTC 
Created attachment 1200801 [details]
File: open_fds
Comment 11 Mikkel Lauritsen 2016-09-14 11:42:17 UTC 
Created attachment 1200802 [details]
File: proc_pid_status
Comment 12 Mikkel Lauritsen 2016-09-14 11:42:18 UTC 
Created attachment 1200803 [details]
File: var_log_messages
Comment 13 Mikkel Lauritsen 2016-09-14 19:27:17 UTC 
The crash happens right away when the smbd service is started. It's apparently independent of the contents of smb.conf - testparam reports that the config is OK, and I've tried starting smb with the default smb.conf with the same result.
Comment 14 Andreas Schneider 2016-09-16 05:31:28 UTC 
Could you please try with https://bodhi.fedoraproject.org/updates/FEDORA-2016-72793a0d3c
Comment 15 Mikkel Lauritsen 2016-09-16 14:56:03 UTC 
Not much of a diffence, unfortunately:

Sep 16 16:51:51 server2.tala.local systemd[1]: Starting Samba SMB Daemon...
Sep 16 16:51:51 server2.tala.local systemd[1]: smb.service: Supervising process 7912 which is not our child. We'll most likely not notice when it exits.
Sep 16 16:51:51 server2.tala.local smbd[7913]: [2016/09/16 16:51:51.464308,  0] ../lib/util/fault.c:78(fault_report)
Sep 16 16:51:51 server2.tala.local smbd[7913]:   ===============================================================
Sep 16 16:51:51 server2.tala.local smbd[7913]: [2016/09/16 16:51:51.465599,  0] ../lib/util/fault.c:79(fault_report)
Sep 16 16:51:51 server2.tala.local smbd[7913]:   INTERNAL ERROR: Signal 11 in pid 7913 (4.5.0)
Sep 16 16:51:51 server2.tala.local smbd[7913]:   Please read the Trouble-Shooting section of the Samba HOWTO
Sep 16 16:51:51 server2.tala.local smbd[7913]: [2016/09/16 16:51:51.467113,  0] ../lib/util/fault.c:81(fault_report)
Sep 16 16:51:51 server2.tala.local smbd[7913]:   ===============================================================
Sep 16 16:51:51 server2.tala.local smbd[7913]: [2016/09/16 16:51:51.468174,  0] ../source3/lib/util.c:791(smb_panic_s3)
Sep 16 16:51:51 server2.tala.local smbd[7913]:   PANIC (pid 7913): internal error
Sep 16 16:51:51 server2.tala.local smbd[7913]: [2016/09/16 16:51:51.469674,  0] ../source3/lib/util.c:902(log_stack_trace)
Sep 16 16:51:51 server2.tala.local smbd[7913]:   BACKTRACE: 14 stack frames:
Sep 16 16:51:51 server2.tala.local smbd[7913]:    #0 /lib64/libsmbconf.so.0(log_stack_trace+0x1c) [0x7fc6b199380c]
Sep 16 16:51:51 server2.tala.local smbd[7913]:    #1 /lib64/libsmbconf.so.0(smb_panic_s3+0x20) [0x7fc6b19938e0]
Sep 16 16:51:51 server2.tala.local smbd[7913]:    #2 /lib64/libsamba-util.so.0(smb_panic+0x2f) [0x7fc6b3e8782f]
Sep 16 16:51:51 server2.tala.local smbd[7913]:    #3 /lib64/libsamba-util.so.0(+0x22a46) [0x7fc6b3e87a46]
Sep 16 16:51:51 server2.tala.local smbd[7913]:    #4 /lib64/libpthread.so.0(+0x115c0) [0x7fc6b40eb5c0]
Sep 16 16:51:51 server2.tala.local smbd[7913]:    #5 /lib64/libtevent.so.0(tevent_debug+0x56) [0x7fc6b03ccf06]
Sep 16 16:51:51 server2.tala.local smbd[7913]:    #6 /lib64/libtevent.so.0(tevent_common_loop_timer_delay+0xba) [0x7fc6b03d14ea]
Sep 16 16:51:51 server2.tala.local smbd[7913]:    #7 /lib64/libsmbconf.so.0(run_events_poll+0x1a9) [0x7fc6b19ab2b9]
Sep 16 16:51:51 server2.tala.local smbd[7913]:    #8 /lib64/libsmbconf.so.0(+0x36457) [0x7fc6b19ab457]
Sep 16 16:51:51 server2.tala.local systemd[1]: Started Samba SMB Daemon.
Sep 16 16:51:51 server2.tala.local smbd[7913]:    #9 /lib64/libtevent.so.0(_tevent_loop_once+0x9d) [0x7fc6b03ccabd]
Sep 16 16:51:51 server2.tala.local smbd[7913]:    #10 /lib64/libtevent.so.0(tevent_req_poll+0x23) [0x7fc6b03cde23]
Sep 16 16:51:51 server2.tala.local smbd[7913]:    #11 /usr/sbin/smbd(main+0x860) [0x55c8653406b0]
Sep 16 16:51:51 server2.tala.local smbd[7913]:    #12 /lib64/libc.so.6(__libc_start_main+0xf1) [0x7fc6b0022401]
Sep 16 16:51:51 server2.tala.local smbd[7913]:    #13 /usr/sbin/smbd(_start+0x2a) [0x55c8653416ba]
Sep 16 16:51:51 server2.tala.local smbd[7913]: [2016/09/16 16:51:51.470044,  0] ../source3/lib/dumpcore.c:303(dump_core)
Sep 16 16:51:51 server2.tala.local smbd[7913]:   dumping core in /var/log/samba/cores/smbd
Sep 16 16:51:51 server2.tala.local smbd[7913]: 
Sep 16 16:51:51 server2.tala.local smbd[7915]: [2016/09/16 16:51:51.471692,  0] ../lib/util/fault.c:78(fault_report)
Sep 16 16:51:51 server2.tala.local smbd[7915]:   ===============================================================
Sep 16 16:51:51 server2.tala.local smbd[7915]: [2016/09/16 16:51:51.471834,  0] ../lib/util/fault.c:79(fault_report)
Sep 16 16:51:51 server2.tala.local smbd[7915]:   INTERNAL ERROR: Signal 11 in pid 7915 (4.5.0)
Sep 16 16:51:51 server2.tala.local smbd[7915]:   Please read the Trouble-Shooting section of the Samba HOWTO
Sep 16 16:51:51 server2.tala.local smbd[7915]: [2016/09/16 16:51:51.471891,  0] ../lib/util/fault.c:81(fault_report)
Sep 16 16:51:51 server2.tala.local smbd[7915]:   ===============================================================
Sep 16 16:51:51 server2.tala.local smbd[7915]: [2016/09/16 16:51:51.471932,  0] ../source3/lib/util.c:791(smb_panic_s3)
Sep 16 16:51:51 server2.tala.local smbd[7915]:   PANIC (pid 7915): internal error
Sep 16 16:51:51 server2.tala.local smbd[7915]: [2016/09/16 16:51:51.472744,  0] ../source3/lib/util.c:902(log_stack_trace)
Sep 16 16:51:51 server2.tala.local smbd[7915]:   BACKTRACE: 5 stack frames:
Sep 16 16:51:51 server2.tala.local smbd[7915]:    #0 /lib64/libsmbconf.so.0(log_stack_trace+0x1c) [0x7fc6b199380c]
Sep 16 16:51:51 server2.tala.local smbd[7915]:    #1 /lib64/libsmbconf.so.0(smb_panic_s3+0x20) [0x7fc6b19938e0]
Sep 16 16:51:51 server2.tala.local smbd[7915]:    #2 /lib64/libsamba-util.so.0(smb_panic+0x2f) [0x7fc6b3e8782f]
Sep 16 16:51:51 server2.tala.local smbd[7915]:    #3 /lib64/libsamba-util.so.0(+0x22a46) [0x7fc6b3e87a46]
Sep 16 16:51:51 server2.tala.local smbd[7915]:    #4 /lib64/libpthread.so.0(+0x115c0) [0x7fc6b40eb5c0]
Sep 16 16:51:51 server2.tala.local smbd[7912]: [2016/09/16 16:51:51.480123,  0] ../lib/util/become_daemon.c:124(daemon_ready)
Sep 16 16:51:51 server2.tala.local smbd[7912]:   STATUS=daemon 'smbd' finished starting up and ready to serve connections
Sep 16 16:51:51 server2.tala.local smbd[7912]: [2016/09/16 16:51:51.540217,  0] ../lib/util/fault.c:78(fault_report)
Sep 16 16:51:51 server2.tala.local smbd[7912]:   ===============================================================
Sep 16 16:51:51 server2.tala.local smbd[7912]: [2016/09/16 16:51:51.540343,  0] ../lib/util/fault.c:79(fault_report)
Sep 16 16:51:51 server2.tala.local smbd[7912]:   INTERNAL ERROR: Signal 11 in pid 7912 (4.5.0)
Sep 16 16:51:51 server2.tala.local smbd[7912]:   Please read the Trouble-Shooting section of the Samba HOWTO
Sep 16 16:51:51 server2.tala.local smbd[7912]: [2016/09/16 16:51:51.540416,  0] ../lib/util/fault.c:81(fault_report)
Sep 16 16:51:51 server2.tala.local smbd[7912]:   ===============================================================
Sep 16 16:51:51 server2.tala.local smbd[7912]: [2016/09/16 16:51:51.540460,  0] ../source3/lib/util.c:791(smb_panic_s3)
Sep 16 16:51:51 server2.tala.local smbd[7912]:   PANIC (pid 7912): internal error
Sep 16 16:51:51 server2.tala.local smbd[7912]: [2016/09/16 16:51:51.540968,  0] ../source3/lib/util.c:902(log_stack_trace)
Sep 16 16:51:51 server2.tala.local smbd[7912]:   BACKTRACE: 19 stack frames:
Sep 16 16:51:51 server2.tala.local smbd[7912]:    #0 /lib64/libsmbconf.so.0(log_stack_trace+0x1c) [0x7fc6b199380c]
Sep 16 16:51:51 server2.tala.local smbd[7912]:    #1 /lib64/libsmbconf.so.0(smb_panic_s3+0x20) [0x7fc6b19938e0]
Sep 16 16:51:51 server2.tala.local smbd[7912]:    #2 /lib64/libsamba-util.so.0(smb_panic+0x2f) [0x7fc6b3e8782f]
Sep 16 16:51:51 server2.tala.local smbd[7912]:    #3 /lib64/libsamba-util.so.0(+0x22a46) [0x7fc6b3e87a46]
Sep 16 16:51:51 server2.tala.local smbd[7912]:    #4 /lib64/libpthread.so.0(+0x115c0) [0x7fc6b40eb5c0]
Sep 16 16:51:51 server2.tala.local smbd[7912]:    #5 /lib64/libtevent.so.0(tevent_timeval_compare+0) [0x7fc6b03d10a0]
Sep 16 16:51:51 server2.tala.local smbd[7912]:    #6 /lib64/libtevent.so.0(+0x9251) [0x7fc6b03d1251]
Sep 16 16:51:51 server2.tala.local smbd[7912]:    #7 /lib64/libtevent.so.0(tevent_common_add_timer+0x13) [0x7fc6b03d1403]
Sep 16 16:51:51 server2.tala.local smbd[7912]:    #8 /lib64/libtevent.so.0(tevent_req_set_endtime+0x60) [0x7fc6b03cdef0]
Sep 16 16:51:51 server2.tala.local smbd[7912]:    #9 /lib64/libtevent.so.0(tevent_wakeup_send+0x55) [0x7fc6b03d17a5]
Sep 16 16:51:51 server2.tala.local smbd[7912]:    #10 /usr/sbin/smbd(+0xaebe) [0x55c865344ebe]
Sep 16 16:51:51 server2.tala.local smbd[7912]:    #11 /lib64/libtevent.so.0(tevent_common_check_signal+0x278) [0x7fc6b03d0a58]
Sep 16 16:51:51 server2.tala.local smbd[7912]:    #12 /lib64/libsmbconf.so.0(run_events_poll+0x24) [0x7fc6b19ab134]
Sep 16 16:51:51 server2.tala.local smbd[7912]:    #13 /lib64/libsmbconf.so.0(+0x364f7) [0x7fc6b19ab4f7]
Sep 16 16:51:51 server2.tala.local smbd[7912]:    #14 /lib64/libtevent.so.0(_tevent_loop_once+0x9d) [0x7fc6b03ccabd]
Sep 16 16:51:51 server2.tala.local smbd[7912]:    #15 /lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7fc6b03ccceb]
Sep 16 16:51:51 server2.tala.local smbd[7912]:    #16 /usr/sbin/smbd(main+0x1642) [0x55c865341492]
Sep 16 16:51:51 server2.tala.local smbd[7912]:    #17 /lib64/libc.so.6(__libc_start_main+0xf1) [0x7fc6b0022401]
Sep 16 16:51:51 server2.tala.local smbd[7912]:    #18 /usr/sbin/smbd(_start+0x2a) [0x55c8653416ba]
Sep 16 16:51:51 server2.tala.local smbd[7912]: [2016/09/16 16:51:51.541426,  0] ../source3/lib/dumpcore.c:303(dump_core)
Sep 16 16:51:51 server2.tala.local smbd[7912]:   dumping core in /var/log/samba/cores/smbd
Comment 16 Andreas Schneider 2016-09-19 06:27:22 UTC 
Could you please post the full backtrace?
Comment 17 Mikkel Lauritsen 2016-09-19 12:16:32 UTC 
Created attachment 1202464 [details]
New backtrace
Comment 18 Andreas Schneider 2016-09-21 11:15:12 UTC 
Could you start smbd with valgrind:

valgrind --tool=memcheck -v --num-callers=20 --track-origins=yes --log-file=smbd-valgrind.log /usr/sbin/smbd

and upload the logfile?
Comment 19 Mikkel Lauritsen 2016-09-22 06:27:46 UTC 
Created attachment 1203601 [details]
Valgrind log file
Comment 20 Andreas Schneider 2016-09-22 07:03:00 UTC 
Could you please run valgrind with debuginfo installed for samba and tevent?
Comment 21 Andreas Schneider 2016-09-22 07:06:55 UTC 
The command is: debuginfo-install samba libtevent
Comment 22 J W 2016-09-22 16:12:34 UTC 
Created attachment 1203844 [details]
Valgrind Log with Debug Info

Does look to be failing in libtevent, something is corrupting the memory because its trying to return to 0xb0 address.  Also, compiling to an earlier version of libtevent seems to fix the problem.
Comment 23 J W 2016-09-24 06:26:59 UTC 
Compiled and installed libtevent-0.9.29 into /usr/local/lib and pointed /usr/lib64/libtevent.so.0 -> /usr/local/lib/libtevent.so.0.9.29 and systemctl restart smb.service works perfectly and I can log in with no Signal 11.
Comment 24 J W 2016-09-24 06:42:37 UTC 
diff of version 0.9.29 and 0.9.30, seems the developer converted a lot of normal code into pthread code. I guess some of that code is corrupting the stack.
Comment 25 Andreas Schneider 2016-09-25 02:06:27 UTC 
The problem is that smbd uses internal tevent structures! The internal structures changed with 0.9.30, but Samba 4.5.x is build with interal structures and has the information of 0.9.29. So accessing the structure leads to segfault.

There is a patchset to not use internal tevent structures in Samba anymore. This will fix the issue. Patches are under review.
Comment 26 Pablo Iranzo Gómez 2016-10-12 21:09:25 UTC 
(In reply to Andreas Schneider from comment #25)
> The problem is that smbd uses internal tevent structures! The internal
> structures changed with 0.9.30, but Samba 4.5.x is build with interal
> structures and has the information of 0.9.29. So accessing the structure
> leads to segfault.
> 
> There is a patchset to not use internal tevent structures in Samba anymore.
> This will fix the issue. Patches are under review.

Hi Andreas
Is there any updates about the patches being reviewed?

Thanks,
Pablo
Comment 27 Andreas Schneider 2016-10-17 13:05:46 UTC 
*** Bug 1384337 has been marked as a duplicate of this bug. ***
Comment 28 Andreas Schneider 2016-10-17 15:05:57 UTC 
*** Bug 1385327 has been marked as a duplicate of this bug. ***
Comment 29 Andreas Schneider 2016-10-17 18:37:34 UTC 
Looks like the bodhi <-> bugzilla conntion is not working ...

https://bodhi.fedoraproject.org/updates/FEDORA-2016-c46eda651e
Comment 30 Fedora Update System 2016-10-19 08:30:20 UTC 
samba-4.5.0-3.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-c46eda651e
Comment 31 Marc Muehlfeld 2016-10-19 15:30:55 UTC 
I can confirm, that samba-4.5.0-3.fc25 fixes the problem here.

Thanks.
Comment 32 Fedora Update System 2016-10-19 17:27:06 UTC 
samba-4.5.0-3.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.
Comment 33 Andreas Schneider 2016-10-26 13:00:49 UTC 
*** Bug 1387517 has been marked as a duplicate of this bug. ***
Comment 34 cube00 2017-06-25 17:28:11 UTC 
Got sent here after reporting a crash in samba-4.5.10-0.fc25.x86_64 but unable to report as it is a duplicate of this closed bug. Any ideas how I can force a new report in abrt?
Comment 35 Andreas Schneider 2017-06-26 09:15:57 UTC 
See comment #31
Comment 36 Nerijus Baliūnas 2017-06-26 09:52:19 UTC 
Comment 31 is about 4.5.0-3.fc25, cube00 is talking about 4.5.10-0.fc25.
Comment 37 cube00 2017-06-27 14:45:26 UTC 
Manually reported in 1465523.
Note You need to log in before you can comment on or make changes to this bug.