Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
For bugs related to Red Hat Enterprise Linux 5 product line. The current stable release is 5.10. For Red Hat Enterprise Linux 6 and above, please visit Red Hat JIRA https://issues.redhat.com/secure/CreateIssue!default.jspa?pid=12332745 to report new issues.

Bug 590610

Summary: Samba3x crashes when logfile rotation occurs
Product: Red Hat Enterprise Linux 5 Reporter: Albert Flügel <albert.fluegel>
Component: samba3xAssignee: Guenther Deschner <gdeschner>
Status: CLOSED NOTABUG QA Contact: qe-baseos-daemons
Severity: high Docs Contact:
Priority: low    
Version: 5.6CC: azelinka, dpal, ssorce
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-06-30 13:41:11 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Albert Flügel 2010-05-10 10:16:54 UTC
Description of problem:

smbd crashes, when the logfiles in /var/log/samba are rotated

Version-Release number of selected component (if applicable):
3.3.8-0.51.el5

How reproducible:
Wait until logfile rotation occurs or force it manually.

Steps to Reproduce:
1. Install and configure samba, so smbd is running
2. Wait until logrotate is rotating the files in /var/log/samba
  
Actual results:
smbd has crashed (no more smbd running).
Sometimes 2 smbd are remaining, but no longer listening on port 139
or whatever is configured. Actually it's unclear, what they are
still doing.

Expected results:
smbd keeps running

Additional info:
in a logfile i see:
[2010/05/09 04:02:05,  0] lib/fault.c:fault_report(40)
  ===============================================================
[2010/05/09 04:02:05,  0] lib/fault.c:fault_report(41)
  INTERNAL ERROR: Signal 7 in pid 14813 (3.3.8-0.51.el5)
  Please read the Trouble-Shooting section of the Samba3-HOWTO
[2010/05/09 04:02:05,  0] lib/fault.c:fault_report(43)
  
  From: http://www.samba.org/samba/docs/Samba3-HOWTO.pdf
[2010/05/09 04:02:05,  0] lib/fault.c:fault_report(44)
  ===============================================================
[2010/05/09 04:02:05,  0] lib/util.c:smb_panic(1673)
  PANIC (pid 14813): internal error
[2010/05/09 04:02:05,  0] lib/util.c:log_stack_trace(1777)
  BACKTRACE: 16 stack frames:
   #0 smbd(log_stack_trace+0x1c) [0x2b5d3076626c]
   #1 smbd(smb_panic+0x2b) [0x2b5d3076634b]
   #2 smbd [0x2b5d3075168e]
   #3 /lib64/libpthread.so.0 [0x2b5d326ce930]
   #4 /lib64/libc.so.6(memcpy+0x35) [0x2b5d3448bde5]
   #5 /usr/lib64/libtdb.so.1 [0x2b5d33ffc4f5]
   #6 /usr/lib64/libtdb.so.1 [0x2b5d33ff8313]
   #7 /usr/lib64/libtdb.so.1 [0x2b5d33ff8422]
   #8 /usr/lib64/libtdb.so.1(tdb_fetch+0x47) [0x2b5d33ff8aa7]
   #9 smbd [0x2b5d3073e032]
   #10 smbd [0x2b5d3073e517]
   #11 smbd(messaging_send+0x46) [0x2b5d3073c636]
   #12 smbd(messaging_send_buf+0x40) [0x2b5d3073c6a0]
   #13 smbd(main+0x118e) [0x2b5d30971c4e]
   #14 /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b5d3442d994]
   #15 smbd [0x2b5d305367c9]
[2010/05/09 04:02:05,  0] lib/fault.c:dump_core(231)
  dumping core in /var/log/samba/cores/smbd
[2010/05/09 04:02:05,  0] lib/fault.c:dump_core(231)
  dumping core in /var/log/samba/cores/smbd
[2010/05/09 04:28:49,  0] lib/fault.c:fault_report(40)
[2010/05/09 04:28:49,  0] lib/fault.c:fault_report(40)
  ===============================================================
  ===============================================================
[2010/05/09 04:28:49,  0] lib/fault.c:fault_report(41)
[2010/05/09 04:28:49,  0] lib/fault.c:fault_report(41)
  INTERNAL ERROR: Signal 7 in pid 14819 (3.3.8-0.51.el5)
  INTERNAL ERROR: Signal 7 in pid 13836 (3.3.8-0.51.el5)
  Please read the Trouble-Shooting section of the Samba3-HOWTO
  Please read the Trouble-Shooting section of the Samba3-HOWTO

Using a debuggable smbd (libtdb is still not debugbable) looking at a core file i find:

Core was generated by `smbd -D'.
Program terminated with signal 6, Aborted.
[New process 1637]
#0  0x00002b0f507c5265 in raise () from /lib64/libc.so.6
(gdb) up
#1  0x00002b0f507c6d10 in abort () from /lib64/libc.so.6
(gdb) up
#2  0x00002b0f4c8c84ad in dump_core () at lib/fault.c:242
up
242		abort();
(gdb) up
#3  0x00002b0f4c8dd6b9 in smb_panic (why=<value optimized out>)
    at lib/util.c:1689
1689		dump_core();
(gdb) up
#4  0x00002b0f4c8c89de in sig_fault (sig=7) at lib/fault.c:46
46		smb_panic("internal error");
(gdb) up
#5  <signal handler called>
(gdb) up
#6  0x00002b0f50810de5 in memcpy () from /lib64/libc.so.6
(gdb) up
#7  0x00002b0f503824f5 in ?? () from /usr/lib64/libtdb.so.1
(gdb) up
#8  0x00002b0f5037e313 in ?? () from /usr/lib64/libtdb.so.1
(gdb) up
#9  0x00002b0f5037e422 in ?? () from /usr/lib64/libtdb.so.1
(gdb) up
#10 0x00002b0f5037eaa7 in tdb_fetch () from /usr/lib64/libtdb.so.1
(gdb) up
#11 0x00002b0f4c8b5382 in messaging_tdb_fetch (msg_tdb=0x2b0f571dddc0, key=
      {dptr = 0x2b0f571e58a0 "PID/1637", dsize = 9}, 
    mem_ctx=<value optimized out>, presult=0x7fffee998538)
    at lib/messages_local.c:153
153		data = tdb_fetch(msg_tdb, key);
(gdb) print key
$1 = {dptr = 0x2b0f571e58a0 "PID/1637", dsize = 9}

The /etc/logrotate.d/samba of the named samba3x versions reads:
cat /etc/logrotate.d/samba 
/var/log/samba/* {
    notifempty
    olddir /var/log/samba/old
    missingok
    sharedscripts
    copytruncate
}

i hope (not verified yet), that adding
    postrotate
       /etc/init.d/smb restart
    endscript
helps.

Comment 4 Guenther Deschner 2010-06-09 16:12:35 UTC
I cannot reproduce this (tried on i386 and x86_64), can you give some more details, what is the version of libtdb you are using ? Can you reproduce it at will (by calling "killall -HUP smbd") ?

Comment 5 Guenther Deschner 2010-06-09 16:14:00 UTC
Also please upload your smb.conf file.

Comment 6 Guenther Deschner 2010-06-17 18:58:02 UTC
Albert, can you give us any hint how to reproduce it please ?

Comment 7 Albert Flügel 2010-06-29 09:13:33 UTC
Hi, sorry i was on vacation.

To reproduce the problem:
cd /var/log/samba
mv messages.tdb old/messages.tdb.0   # or whatever name you like

Now initiate some new samba connection or perform any other action making smbd log into the messages file.
The messages.tdb is not always rotated by logrotate it seems, so simply calling the log rotater is not sufficient to reproduce the problem.

As soon as you see a recreated messages.tdb, you will see the core dump message in log.0.0.0.0

the libtdb version is libtdb-1.1.2-51.el5

Please try to reproduce the described way. I'd like to only upload the smbd.conf file, if unavoidable to see the problem.

hope this helps.

Comment 8 Guenther Deschner 2010-06-30 10:56:39 UTC
Oh, I see. 

So, messages.tdb does *NOT* contain logging information but is an internal database that is used to store interprocess messaging information. It is illegal to remove it underneath running samba daemons. logrotate will and must never ever touch it.

Actually it can only happen if you have set "lock directory = /var/log/samba" smb.conf option. Have you set that ?

If so, can you try this patch?

--- /etc/logrotate.d/samba.orig 2010-06-30 12:40:02.863610180 +0200
+++ /etc/logrotate.d/samba      2010-06-30 12:40:08.749275101 +0200
@@ -1,4 +1,4 @@
-/var/log/samba/* {
+/var/log/samba/*.log {
     notifempty
     olddir /var/log/samba/old
     missingok

Comment 9 Albert Flügel 2010-06-30 11:10:43 UTC
The /etc/logrotate.d/samba came with samba3x-3.3.8-0.51.el5 and looks like this:
/var/log/samba/* {
    notifempty
    olddir /var/log/samba/old
    missingok
    sharedscripts
    copytruncate
}

I added

>     postrotate
>        /etc/init.d/smb restart
>     endscript

to work around the problem.

i think your patch will do the job and avoid rotating .tdb files. Can you confirm, that the .tdb files are not continuously growing ?
Thanks a lot !

Things are more or less like i suspected.
The reason why i opened the bug against samba3x is, that the logrotate.d/samba
file is part of the respective rpm.

Comment 10 Albert Flügel 2010-06-30 11:15:27 UTC
erm, more log files to be rotated are
log.<ip-address>

... think your pattern is not sufficient.
Will try with

/var/log/samba/*.log /var/log/samba/log.* {
    notifempty
    olddir /var/log/samba/old
    missingok
    sharedscripts
    copytruncate
}

Comment 11 Guenther Deschner 2010-06-30 13:38:55 UTC
(In reply to comment #9)
> The /etc/logrotate.d/samba came with samba3x-3.3.8-0.51.el5 and looks like
> this:
> /var/log/samba/* {
>     notifempty
>     olddir /var/log/samba/old
>     missingok
>     sharedscripts
>     copytruncate
> }
> 
> I added
> 
> >     postrotate
> >        /etc/init.d/smb restart
> >     endscript
> 
> to work around the problem.

No, this workaround is really wrong as it kills your smbd during logrotate, including disconnects of all existing connections.

> i think your patch will do the job and avoid rotating .tdb files. Can you
> confirm, that the .tdb files are not continuously growing ?

Usually they shouldn't, if you observe one to grow to a problematic degree, please open a new bug.

Note that just copying an opened tdb file is the wrong way (and therefore logrotate cannot create working or usefull copies). For creating backups of tdbs, please use tdbbackup.

Comment 12 Guenther Deschner 2010-06-30 13:41:11 UTC
(In reply to comment #10)
> erm, more log files to be rotated are
> log.<ip-address>
> 
> ... think your pattern is not sufficient.
> Will try with
> 
> /var/log/samba/*.log /var/log/samba/log.* {
>     notifempty
>     olddir /var/log/samba/old
>     missingok
>     sharedscripts
>     copytruncate
> }    

Yeah, depending on other options in your smb.conf file, other matching patterns need to be used. Really hard to match whatever is set in smb.conf :)

I think we can close this as not a bug, as default setups will work just fine with logrotate.

Thanks for the detailed report though!

Comment 13 Albert Flügel 2010-06-30 14:29:15 UTC
Sorry, i disagree. The .tdb files are matched by /var/log/samba/* and thus rotated, what will cause the core dump, so i consider the file pattern in /etc/logrotate.d/samba a bug.

Comment 14 Simo Sorce 2010-06-30 14:57:30 UTC
Albert,
sorry but changing the lockdir directory in smb.conf to point to /var/log/samba  (or symlinking /var/lib/samba to it) is not supported.

All files in /var/log/samba/ are considered log files and they are all rotated without discrimination on purpose.

We consider this a configuration mistake and not a bug, sorry.