Red Hat Bugzilla – Bug 590610
Samba3x crashes when logfile rotation occurs
Last modified: 2010-11-09 08:15:36 EST
Description of problem: smbd crashes, when the logfiles in /var/log/samba are rotated Version-Release number of selected component (if applicable): 3.3.8-0.51.el5 How reproducible: Wait until logfile rotation occurs or force it manually. Steps to Reproduce: 1. Install and configure samba, so smbd is running 2. Wait until logrotate is rotating the files in /var/log/samba Actual results: smbd has crashed (no more smbd running). Sometimes 2 smbd are remaining, but no longer listening on port 139 or whatever is configured. Actually it's unclear, what they are still doing. Expected results: smbd keeps running Additional info: in a logfile i see: [2010/05/09 04:02:05, 0] lib/fault.c:fault_report(40) =============================================================== [2010/05/09 04:02:05, 0] lib/fault.c:fault_report(41) INTERNAL ERROR: Signal 7 in pid 14813 (3.3.8-0.51.el5) Please read the Trouble-Shooting section of the Samba3-HOWTO [2010/05/09 04:02:05, 0] lib/fault.c:fault_report(43) From: http://www.samba.org/samba/docs/Samba3-HOWTO.pdf [2010/05/09 04:02:05, 0] lib/fault.c:fault_report(44) =============================================================== [2010/05/09 04:02:05, 0] lib/util.c:smb_panic(1673) PANIC (pid 14813): internal error [2010/05/09 04:02:05, 0] lib/util.c:log_stack_trace(1777) BACKTRACE: 16 stack frames: #0 smbd(log_stack_trace+0x1c) [0x2b5d3076626c] #1 smbd(smb_panic+0x2b) [0x2b5d3076634b] #2 smbd [0x2b5d3075168e] #3 /lib64/libpthread.so.0 [0x2b5d326ce930] #4 /lib64/libc.so.6(memcpy+0x35) [0x2b5d3448bde5] #5 /usr/lib64/libtdb.so.1 [0x2b5d33ffc4f5] #6 /usr/lib64/libtdb.so.1 [0x2b5d33ff8313] #7 /usr/lib64/libtdb.so.1 [0x2b5d33ff8422] #8 /usr/lib64/libtdb.so.1(tdb_fetch+0x47) [0x2b5d33ff8aa7] #9 smbd [0x2b5d3073e032] #10 smbd [0x2b5d3073e517] #11 smbd(messaging_send+0x46) [0x2b5d3073c636] #12 smbd(messaging_send_buf+0x40) [0x2b5d3073c6a0] #13 smbd(main+0x118e) [0x2b5d30971c4e] #14 /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b5d3442d994] #15 smbd [0x2b5d305367c9] [2010/05/09 04:02:05, 0] lib/fault.c:dump_core(231) dumping core in /var/log/samba/cores/smbd [2010/05/09 04:02:05, 0] lib/fault.c:dump_core(231) dumping core in /var/log/samba/cores/smbd [2010/05/09 04:28:49, 0] lib/fault.c:fault_report(40) [2010/05/09 04:28:49, 0] lib/fault.c:fault_report(40) =============================================================== =============================================================== [2010/05/09 04:28:49, 0] lib/fault.c:fault_report(41) [2010/05/09 04:28:49, 0] lib/fault.c:fault_report(41) INTERNAL ERROR: Signal 7 in pid 14819 (3.3.8-0.51.el5) INTERNAL ERROR: Signal 7 in pid 13836 (3.3.8-0.51.el5) Please read the Trouble-Shooting section of the Samba3-HOWTO Please read the Trouble-Shooting section of the Samba3-HOWTO Using a debuggable smbd (libtdb is still not debugbable) looking at a core file i find: Core was generated by `smbd -D'. Program terminated with signal 6, Aborted. [New process 1637] #0 0x00002b0f507c5265 in raise () from /lib64/libc.so.6 (gdb) up #1 0x00002b0f507c6d10 in abort () from /lib64/libc.so.6 (gdb) up #2 0x00002b0f4c8c84ad in dump_core () at lib/fault.c:242 up 242 abort(); (gdb) up #3 0x00002b0f4c8dd6b9 in smb_panic (why=<value optimized out>) at lib/util.c:1689 1689 dump_core(); (gdb) up #4 0x00002b0f4c8c89de in sig_fault (sig=7) at lib/fault.c:46 46 smb_panic("internal error"); (gdb) up #5 <signal handler called> (gdb) up #6 0x00002b0f50810de5 in memcpy () from /lib64/libc.so.6 (gdb) up #7 0x00002b0f503824f5 in ?? () from /usr/lib64/libtdb.so.1 (gdb) up #8 0x00002b0f5037e313 in ?? () from /usr/lib64/libtdb.so.1 (gdb) up #9 0x00002b0f5037e422 in ?? () from /usr/lib64/libtdb.so.1 (gdb) up #10 0x00002b0f5037eaa7 in tdb_fetch () from /usr/lib64/libtdb.so.1 (gdb) up #11 0x00002b0f4c8b5382 in messaging_tdb_fetch (msg_tdb=0x2b0f571dddc0, key= {dptr = 0x2b0f571e58a0 "PID/1637", dsize = 9}, mem_ctx=<value optimized out>, presult=0x7fffee998538) at lib/messages_local.c:153 153 data = tdb_fetch(msg_tdb, key); (gdb) print key $1 = {dptr = 0x2b0f571e58a0 "PID/1637", dsize = 9} The /etc/logrotate.d/samba of the named samba3x versions reads: cat /etc/logrotate.d/samba /var/log/samba/* { notifempty olddir /var/log/samba/old missingok sharedscripts copytruncate } i hope (not verified yet), that adding postrotate /etc/init.d/smb restart endscript helps.
I cannot reproduce this (tried on i386 and x86_64), can you give some more details, what is the version of libtdb you are using ? Can you reproduce it at will (by calling "killall -HUP smbd") ?
Also please upload your smb.conf file.
Albert, can you give us any hint how to reproduce it please ?
Hi, sorry i was on vacation. To reproduce the problem: cd /var/log/samba mv messages.tdb old/messages.tdb.0 # or whatever name you like Now initiate some new samba connection or perform any other action making smbd log into the messages file. The messages.tdb is not always rotated by logrotate it seems, so simply calling the log rotater is not sufficient to reproduce the problem. As soon as you see a recreated messages.tdb, you will see the core dump message in log.0.0.0.0 the libtdb version is libtdb-1.1.2-51.el5 Please try to reproduce the described way. I'd like to only upload the smbd.conf file, if unavoidable to see the problem. hope this helps.
Oh, I see. So, messages.tdb does *NOT* contain logging information but is an internal database that is used to store interprocess messaging information. It is illegal to remove it underneath running samba daemons. logrotate will and must never ever touch it. Actually it can only happen if you have set "lock directory = /var/log/samba" smb.conf option. Have you set that ? If so, can you try this patch? --- /etc/logrotate.d/samba.orig 2010-06-30 12:40:02.863610180 +0200 +++ /etc/logrotate.d/samba 2010-06-30 12:40:08.749275101 +0200 @@ -1,4 +1,4 @@ -/var/log/samba/* { +/var/log/samba/*.log { notifempty olddir /var/log/samba/old missingok
The /etc/logrotate.d/samba came with samba3x-3.3.8-0.51.el5 and looks like this: /var/log/samba/* { notifempty olddir /var/log/samba/old missingok sharedscripts copytruncate } I added > postrotate > /etc/init.d/smb restart > endscript to work around the problem. i think your patch will do the job and avoid rotating .tdb files. Can you confirm, that the .tdb files are not continuously growing ? Thanks a lot ! Things are more or less like i suspected. The reason why i opened the bug against samba3x is, that the logrotate.d/samba file is part of the respective rpm.
erm, more log files to be rotated are log.<ip-address> ... think your pattern is not sufficient. Will try with /var/log/samba/*.log /var/log/samba/log.* { notifempty olddir /var/log/samba/old missingok sharedscripts copytruncate }
(In reply to comment #9) > The /etc/logrotate.d/samba came with samba3x-3.3.8-0.51.el5 and looks like > this: > /var/log/samba/* { > notifempty > olddir /var/log/samba/old > missingok > sharedscripts > copytruncate > } > > I added > > > postrotate > > /etc/init.d/smb restart > > endscript > > to work around the problem. No, this workaround is really wrong as it kills your smbd during logrotate, including disconnects of all existing connections. > i think your patch will do the job and avoid rotating .tdb files. Can you > confirm, that the .tdb files are not continuously growing ? Usually they shouldn't, if you observe one to grow to a problematic degree, please open a new bug. Note that just copying an opened tdb file is the wrong way (and therefore logrotate cannot create working or usefull copies). For creating backups of tdbs, please use tdbbackup.
(In reply to comment #10) > erm, more log files to be rotated are > log.<ip-address> > > ... think your pattern is not sufficient. > Will try with > > /var/log/samba/*.log /var/log/samba/log.* { > notifempty > olddir /var/log/samba/old > missingok > sharedscripts > copytruncate > } Yeah, depending on other options in your smb.conf file, other matching patterns need to be used. Really hard to match whatever is set in smb.conf :) I think we can close this as not a bug, as default setups will work just fine with logrotate. Thanks for the detailed report though!
Sorry, i disagree. The .tdb files are matched by /var/log/samba/* and thus rotated, what will cause the core dump, so i consider the file pattern in /etc/logrotate.d/samba a bug.
Albert, sorry but changing the lockdir directory in smb.conf to point to /var/log/samba (or symlinking /var/lib/samba to it) is not supported. All files in /var/log/samba/ are considered log files and they are all rotated without discrimination on purpose. We consider this a configuration mistake and not a bug, sorry.