Bug 830331

Summary: ns-slapd exits/crashes if /var fills up
Product: Red Hat Enterprise Linux 6 Reporter: Nathan Kinder <nkinder>
Component: 389-ds-baseAssignee: Rich Megginson <rmeggins>
Status: CLOSED ERRATA QA Contact: Sankar Ramalingam <sramling>
Severity: unspecified Docs Contact:
Priority: medium    
Version: 6.4CC: jgalipea, jrusnack, mreynolds
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 389-ds-base-1.2.11.12-1.el6 Doc Type: Enhancement
Doc Text:
Feature: Disk Monitoring Reason: If certain disk partitions fill up, the server should take action to free up space, and/or shutdown cleanly. Result (if any): DS does not crash when disk fills up.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-21 08:17:21 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Nathan Kinder 2012-06-08 21:06:48 UTC 
This bug is created as a clone of upstream ticket:
https://fedorahosted.org/389/ticket/315

I'm not sure if anything can be done here, but currently ns-slapd will crash/exit if /var fills up.  Now the database is mainly read-only so it seems that the only thing that would be written to would be the access log.  I would rather that the server stay active and loose access log entries than have the server go down.
Comment 1 RHEL Program Management 2012-07-10 06:18:18 UTC 
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.
Comment 2 RHEL Program Management 2012-07-10 22:57:33 UTC 
This request was erroneously removed from consideration in Red Hat Enterprise Linux 6.4, which is currently under development.  This request will be evaluated for inclusion in Red Hat Enterprise Linux 6.4.
Comment 4 Sankar Ramalingam 2013-01-24 12:00:07 UTC 
Request you to please add verification steps.
Comment 5 mreynolds 2013-01-24 14:24:04 UTC 
Here is the design doc:

http://port389.org/wiki/Disk_Monitoring

Note, this is a new feature, not a bug fix.

To do a basic test set this in cn=config

nsslapd-disk-monitioring: on
nsslapd-disk-monitoring-threshold: ????  <value in bytes>
nsslapd-disk-monitoring-grace-period: 1 <one minute>
nsslapd-disk-monitoring-logging_critical: on

The idea is that if we get too low on disk space the DS will shutdown itself down gracefully.

What triggers the shutdown is when we get to less than half of threshold, and we exceed the grace period.

So if the available disk space is 8 gigs, and we set the threshold to 4 gigs, then when the available disk space gets below 2 gigs we goto into "shutdown mode".  The server will wait for the length of the grace period (in this example it's 1 minute), and if the available disk space is still less than half of the threshold we shutdown the server.

As for the treshold value, you need to know how much availalble space is on the test machine, and then set a value thats sufficient enough to trigger the issue.

To be honest there are many different scenarios to test, but this will test the basic functionality.
Comment 6 mreynolds 2013-01-24 15:29:54 UTC 
I forgot to mention, you need to restart the DS after setting the config attributes.
Comment 7 Ján Rusnačko 2013-01-30 10:38:33 UTC 
[jrusnack@dstet ~]$ ldapsearch -h localhost -p 22222 -D "cn=directory manager " -w Secret123 -LLL -b "cn=config" | grep disk-monitoring
nsslapd-disk-monitoring: on
nsslapd-disk-monitoring-threshold: 2000000000
nsslapd-disk-monitoring-grace-period: 1
nsslapd-disk-monitoring-logging-critical: on
nsslapd-disk-monitoring-preserve-logging: off

[jrusnack@dstet ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_dstet-lv_root
                       13G   10G  1.8G  86% /
tmpfs                 499M  4.0K  499M   1% /dev/shm
/dev/vda1             485M   77M  383M  17% /boot

[jrusnack@dstet ~]$ tail /var/log/dirsrv/slapd-dstet/errors
...
[30/Jan/2013:05:28:28 -0500] - Disk space is too low on disk (/), remaining space: 1798232 Kb, disabling access and audit logging
...

Fill up the disk space...

[jrusnack@dstet ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_dstet-lv_root
                       13G   12G  444M  97% /
tmpfs                 499M  4.0K  499M   1% /dev/shm
/dev/vda1             485M   77M  383M  17% /boot
[jrusnack@dstet ~]$ tail /var/log/dirsrv/slapd-dstet/errors
[30/Jan/2013:05:34:02 -0500] - Disk space is now within acceptable levels.
[30/Jan/2013:05:35:16 -0500] - Disk space is too low on disk (/), remaining space: 976316 Kb, disabling access and audit logging.
[30/Jan/2013:05:35:27 -0500] - Disk space is too low on disk (/), remaining space: 454208 Kb, deleting rotated logs.
[30/Jan/2013:05:35:37 -0500] - Disk space on (/) is too far below the threshold(2000000000 bytes).  Waiting 1 minutes for disk space to be cleaned up before shutting slapd down...
[30/Jan/2013:05:36:37 -0500] - Disk space is still too low (454208 Kb).  Signaling slapd for shutdown...
[30/Jan/2013:05:36:37 -0500] - slapd shutting down - signaling operation threads
[30/Jan/2013:05:36:37 -0500] - slapd shutting down - closing down internal subsystems and plugins
[30/Jan/2013:05:36:37 -0500] - Waiting for 4 database threads to stop
[30/Jan/2013:05:36:37 -0500] - All database threads now stopped
[30/Jan/2013:05:36:37 -0500] - slapd stopped.

Verified.
Comment 9 errata-xmlrpc 2013-02-21 08:17:21 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0503.html