Bug 668385

Summary: DS pipe log script is executed as many times as the dirsrv service is restarted
Product: [Retired] 389 Reporter: Sankar Ramalingam <sramling>
Component: Command Line UtilitiesAssignee: Rich Megginson <rmeggins>
Status: CLOSED CURRENTRELEASE QA Contact: Viktor Ashirov <vashirov>
Severity: medium Docs Contact:
Priority: high    
Version: 1.2.8CC: rmeggins
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 676666 (view as bug list) Environment:
Last Closed: 2015-12-07 16:47:36 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 639035, 656390, 676666    
Attachments:
Description Flags
0001-Bug-668385-DS-pipe-log-script-is-executed-as-many-ti.patch
nkinder: review+
0001-Bug-668385-DS-pipe-log-script-is-executed-as-many-ti.patch nkinder: review+

Description Sankar Ramalingam 2011-01-10 11:10:53 UTC
Description of problem: 
DS pipe log script is getting executed as many times as the service is restarted.
This happens when you place the ds-logpipe.py script in the /etc/sysconfig/dirsrv-$inst_name file to start the DS pipelog script when the instance is started.


Version-Release number of selected component (if applicable): DS90


How reproducible: Consistently


Steps to Reproduce:
1. Configure access logs for named pipe. Set these access log attributes using
ldapmodify.
    nsslapd-accesslog-maxlogsperdir: 1
    nsslapd-accesslog-logexpirationtime: -1
    nsslapd-accesslog-logrotationtime: -1
    nsslapd-accesslog: /var/log/dirsrv/slapd-$inst_name/access.pipe
    nsslapd-accesslog-logbuffering: off

2. Append the below line into the /etc/sysconfig/dirsrv-$inst_name file.
    /usr/bin/ds-logpipe.py /var/log/dirsrv/slapd-inst/access.pipe
--plugin=/usr/share/dirsrv/data/logregex.py logregex.regex="err=32" >
/tmp/test_access &
3. Restart the DS instance using the "service restart dirsrv" command
4. Make sure the ds-logpipe.py script is started after running the above command.
5. Restart the dirsrv service one more time.
6. ds-logpipe.py is being executed everytime the service restarted.  

Actual results:
DS pipe log script is getting executed as many times as the service is restarted.

Expected results: It is expected to run only one instance of the DS pipe log script.

Comment 1 Rich Megginson 2011-02-08 03:11:42 UTC
Created attachment 477547 [details]
0001-Bug-668385-DS-pipe-log-script-is-executed-as-many-ti.patch

Comment 2 Rich Megginson 2011-02-10 16:47:37 UTC
To ssh://git.fedorahosted.org/git/389/ds.git
   30cb812..1132e09  master -> master
commit 1132e09752c8f9aba7a963bfc91a8941000012ca
Author: Rich Megginson <rmeggins@redhat.com>
Date:   Thu Feb 3 18:42:37 2011 -0700
    Reviewed by: nkinder (Thanks!)
    Branch: master
    Fix Description: The main thing was adding a pid file for the script itself,
    so it can know if it is running or not.  This is the new '-i' argument.
    The other tricky part was figuring out, when reading the pipe, it is hard to
    know if the server has really exited or not, and therefore, if the script
    can exit.  When the server is starting up, when it daemonizes, it will close
    and reopen the log file descriptors.  We can detect this as reading just
    an eof (line == None) from the pipe and nothing else.  In this case, we
    can just reopen the log.
    When we open the log file, if reading from a server, we always set a timer
    because the open will block until something writes to the pipe.  However,
    readline() will always return and will never block.  Even at eof, if the
    other side has closed the pipe, readline() will just return None.
    The documentation on the wiki has been updated with the new information
    about how to set up the init scripts to use the logpipe.
    Platforms tested: RHEL6 x86_64
    Flag Day: no
    Doc impact: no
To ssh://git.fedorahosted.org/git/389/ds.git
   f0e39fd..ceec7ba  389-ds-base-1.2.8 -> 389-ds-base-1.2.8
commit ceec7ba4e3b9d8baadf2a34591658484606fab9b
Author: Rich Megginson <rmeggins@redhat.com>
Date:   Thu Feb 3 18:42:37 2011 -0700

Comment 3 Sankar Ramalingam 2011-03-28 14:49:17 UTC
I encountered few more problems when I was testing the fix.

1. Restart/Stop the slapd instance when the DS pipe log script is running.

Result: DS pipe log script is killed and the DS fails to start.

2. When DS is stopped, run the DS pipe log script and then start the DS instance.

Result: DS pipe log script is killed and the DS fails to start.

3. Put the DS pipe log script in the startup script(/etc/sysconfig/dirsrv-$inst).

Result: Instance failed to restart/start when the DS pipe log script is already running/not running.

Ultimately, this fix caused many problems and I also see that the pipe log script is failing redirect the error messages to the pipe file.

Comment 4 Rich Megginson 2011-03-28 14:55:54 UTC
(In reply to comment #3)
> I encountered few more problems when I was testing the fix.
> 
> 1. Restart/Stop the slapd instance when the DS pipe log script is running.
> 
> Result: DS pipe log script is killed and the DS fails to start.

In this case, do you have the DS log pipe script configured to start in the /etc/sysconfig/dirsrv-instance file?  Can you provide the exact steps and configuration to reproduce this problem?

> 
> 2. When DS is stopped, run the DS pipe log script and then start the DS
> instance.
> 
> Result: DS pipe log script is killed and the DS fails to start.

If you have the log pipe script configured to start in the /etc/sysconfig/dirsrv-instance file, then you cannot also run the DS log pipe script separately for the same log file.

> 
> 3. Put the DS pipe log script in the startup
> script(/etc/sysconfig/dirsrv-$inst).
> 
> Result: Instance failed to restart/start when the DS pipe log script is already
> running/not running.
> 
> Ultimately, this fix caused many problems and I also see that the pipe log
> script is failing redirect the error messages to the pipe file.

Can you provide the exact configuration and steps to reproduce?  Note that if you have the DS log pipe configured in /etc/sysconfig/dirsrv-$inst, you cannot manually run the DS log pipe - it must be started and stopped with the directory server using service or the stop/start/restart-slapd scripts.

Comment 5 Sankar Ramalingam 2011-03-29 05:58:08 UTC
(In reply to comment #4)
> (In reply to comment #3)
> > I encountered few more problems when I was testing the fix.
> > 
> > 1. Restart/Stop the slapd instance when the DS pipe log script is running.
> > 
> > Result: DS pipe log script is killed and the DS fails to start.
> 
> In this case, do you have the DS log pipe script configured to start in the
> /etc/sysconfig/dirsrv-instance file?  Can you provide the exact steps and
> configuration to reproduce this problem?
> 
Steps to reproduce:
1. Create the directory server instance and change the "cn=config" entry for the named pipe log.
2. Remove the access log file from /var/log/dirsrv/slapd-$inst/.
3. Run /usr/bin/ds-logpipe.py /var/log/dirsrv/slapd-$inst/access -t 60 --plugin=/usr/share/dirsrv/data/logregex.py logregex.regex="err="
4. Restart the directory server instance to apply the (cn=config) changes.
5. Server fails to restart since the ds-logpipe.py script is killed when stopping the server.

6. Server can be brought back alive only if I re-run the ds-logpipe.py script or if I remove the named pipe file.

> > 
> > 2. When DS is stopped, run the DS pipe log script and then start the DS
> > instance.
> > 
> > Result: DS pipe log script is killed and the DS fails to start.
> 
> If you have the log pipe script configured to start in the
> /etc/sysconfig/dirsrv-instance file, then you cannot also run the DS log pipe
> script separately for the same log file.
> 
I do not have the log pipe script configured in the /etc/sysconfig/dirsrv-instance file.
> > 
> > 3. Put the DS pipe log script in the startup
> > script(/etc/sysconfig/dirsrv-$inst).
> > 
> > Result: Instance failed to restart/start when the DS pipe log script is already
> > running/not running.
> > 
> > Ultimately, this fix caused many problems and I also see that the pipe log
> > script is failing redirect the error messages to the pipe file.
> 
> Can you provide the exact configuration and steps to reproduce?  Note that if
> you have the DS log pipe configured in /etc/sysconfig/dirsrv-$inst, you cannot
> manually run the DS log pipe - it must be started and stopped with the
> directory server using service or the stop/start/restart-slapd scripts.

I am using the service command to stop/start/restart the dirsrv instances.

Comment 6 Rich Megginson 2011-03-29 16:53:53 UTC
(In reply to comment #5)
> (In reply to comment #4)
> > (In reply to comment #3)
> > > I encountered few more problems when I was testing the fix.
> > > 
> > > 1. Restart/Stop the slapd instance when the DS pipe log script is running.
> > > 
> > > Result: DS pipe log script is killed and the DS fails to start.
> > 
> > In this case, do you have the DS log pipe script configured to start in the
> > /etc/sysconfig/dirsrv-instance file?  Can you provide the exact steps and
> > configuration to reproduce this problem?
> > 
> Steps to reproduce:
> 1. Create the directory server instance and change the "cn=config" entry for
> the named pipe log.
> 2. Remove the access log file from /var/log/dirsrv/slapd-$inst/.
> 3. Run /usr/bin/ds-logpipe.py /var/log/dirsrv/slapd-$inst/access -t 60
> --plugin=/usr/share/dirsrv/data/logregex.py logregex.regex="err="

-t only applies when using -s or --serverpid - otherwise it does nothing - please file a bug to update the script help, man page, and documentation 

> 4. Restart the directory server instance to apply the (cn=config) changes.
> 5. Server fails to restart since the ds-logpipe.py script is killed when
> stopping the server.
> 
> 6. Server can be brought back alive only if I re-run the ds-logpipe.py script
> or if I remove the named pipe file.

Ok.  This is definitely a bug.

> 
> > > 
> > > 2. When DS is stopped, run the DS pipe log script and then start the DS
> > > instance.
> > > 
> > > Result: DS pipe log script is killed and the DS fails to start.
> > 
> > If you have the log pipe script configured to start in the
> > /etc/sysconfig/dirsrv-instance file, then you cannot also run the DS log pipe
> > script separately for the same log file.
> > 
> I do not have the log pipe script configured in the
> /etc/sysconfig/dirsrv-instance file.

Ok.  This is the same bug - the script has the same problem with start and with restart. 

> > > 
> > > 3. Put the DS pipe log script in the startup
> > > script(/etc/sysconfig/dirsrv-$inst).
> > > 
> > > Result: Instance failed to restart/start when the DS pipe log script is already
> > > running/not running.
> > > 
> > > Ultimately, this fix caused many problems and I also see that the pipe log
> > > script is failing redirect the error messages to the pipe file.
> > 
> > Can you provide the exact configuration and steps to reproduce?  Note that if
> > you have the DS log pipe configured in /etc/sysconfig/dirsrv-$inst, you cannot
> > manually run the DS log pipe - it must be started and stopped with the
> > directory server using service or the stop/start/restart-slapd scripts.
> 
> I am using the service command to stop/start/restart the dirsrv instances.

Is the problem that you run ds-logpipe first, then configure the ds-logpipe in /etc/sysconfig/dirsrv-$inst, then try to start the server?  That won't work.  You can't mix running ds-logpipe manually _and_ having it run from /etc/sysconfig/dirsrv-$inst

Comment 7 Rich Megginson 2011-03-29 20:10:41 UTC
Created attachment 488552 [details]
0001-Bug-668385-DS-pipe-log-script-is-executed-as-many-ti.patch

Comment 8 Rich Megginson 2011-03-29 21:13:36 UTC
To ssh://git.fedorahosted.org/git/389/ds.git
   0699244..bb02e98  master -> master
commit bb02e98ee264fe37f4ab6495872a470170277b12
Author: Rich Megginson <rmeggins@redhat.com>
Date:   Tue Mar 29 14:09:02 2011 -0600
389-ds-base-1.2.8 branch
commit 51788359843fb9da1b5572c31b25822164462aa4
Author: Rich Megginson <rmeggins@redhat.com>
Date:   Tue Mar 29 14:09:02 2011 -0600

Comment 9 Sankar Ramalingam 2011-03-30 05:08:06 UTC
(In reply to comment #6)
> (In reply to comment #5)
> > (In reply to comment #4)
> > > (In reply to comment #3)
> > > > I encountered few more problems when I was testing the fix.
> > > > 
> > > > 1. Restart/Stop the slapd instance when the DS pipe log script is running.
> > > > 
> > > > Result: DS pipe log script is killed and the DS fails to start.
> > > 
> > > In this case, do you have the DS log pipe script configured to start in the
> > > /etc/sysconfig/dirsrv-instance file?  Can you provide the exact steps and
> > > configuration to reproduce this problem?
> > > 
> > Steps to reproduce:
> > 1. Create the directory server instance and change the "cn=config" entry for
> > the named pipe log.
> > 2. Remove the access log file from /var/log/dirsrv/slapd-$inst/.
> > 3. Run /usr/bin/ds-logpipe.py /var/log/dirsrv/slapd-$inst/access -t 60
> > --plugin=/usr/share/dirsrv/data/logregex.py logregex.regex="err="
> 
> -t only applies when using -s or --serverpid - otherwise it does nothing -
> please file a bug to update the script help, man page, and documentation 
> 
Created a new bug for this. Bug #691980.
> > 4. Restart the directory server instance to apply the (cn=config) changes.
> > 5. Server fails to restart since the ds-logpipe.py script is killed when
> > stopping the server.
> > 
> > 6. Server can be brought back alive only if I re-run the ds-logpipe.py script
> > or if I remove the named pipe file.
> 
> Ok.  This is definitely a bug.
> 
I am not creating a new bug for this since its a regression.
> > 
> > > > 
> > > > 2. When DS is stopped, run the DS pipe log script and then start the DS
> > > > instance.
> > > > 
> > > > Result: DS pipe log script is killed and the DS fails to start.
> > > 
> > > If you have the log pipe script configured to start in the
> > > /etc/sysconfig/dirsrv-instance file, then you cannot also run the DS log pipe
> > > script separately for the same log file.
> > > 
> > I do not have the log pipe script configured in the
> > /etc/sysconfig/dirsrv-instance file.
> 
> Ok.  This is the same bug - the script has the same problem with start and with
> restart. 
> 
> > > > 
> > > > 3. Put the DS pipe log script in the startup
> > > > script(/etc/sysconfig/dirsrv-$inst).
> > > > 
> > > > Result: Instance failed to restart/start when the DS pipe log script is already
> > > > running/not running.
> > > > 
> > > > Ultimately, this fix caused many problems and I also see that the pipe log
> > > > script is failing redirect the error messages to the pipe file.
> > > 
> > > Can you provide the exact configuration and steps to reproduce?  Note that if
> > > you have the DS log pipe configured in /etc/sysconfig/dirsrv-$inst, you cannot
> > > manually run the DS log pipe - it must be started and stopped with the
> > > directory server using service or the stop/start/restart-slapd scripts.
> > 
> > I am using the service command to stop/start/restart the dirsrv instances.
> 
> Is the problem that you run ds-logpipe first, then configure the ds-logpipe in
> /etc/sysconfig/dirsrv-$inst, then try to start the server?  That won't work. 
> You can't mix running ds-logpipe manually _and_ having it run from
> /etc/sysconfig/dirsrv-$inst
Yes, I ran the ds-logpipe first and then configured the startup script. But, later I found that configuring only the startup script was also not working.
Anyways, let me verify the same once I get the new build.

Comment 10 Sankar Ramalingam 2011-05-18 14:41:07 UTC
Fix verified with the RHEL6.1 0day errata build of 389-ds-base.