Bug 830331 - ns-slapd exits/crashes if /var fills up
ns-slapd exits/crashes if /var fills up
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: 389-ds-base (Show other bugs)
Unspecified Unspecified
medium Severity unspecified
: rc
: ---
Assigned To: Rich Megginson
Sankar Ramalingam
Depends On:
  Show dependency treegraph
Reported: 2012-06-08 17:06 EDT by Nathan Kinder
Modified: 2013-07-23 07:24 EDT (History)
3 users (show)

See Also:
Fixed In Version: 389-ds-base-
Doc Type: Enhancement
Doc Text:
Feature: Disk Monitoring Reason: If certain disk partitions fill up, the server should take action to free up space, and/or shutdown cleanly. Result (if any): DS does not crash when disk fills up.
Story Points: ---
Clone Of:
Last Closed: 2013-02-21 03:17:21 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Nathan Kinder 2012-06-08 17:06:48 EDT
This bug is created as a clone of upstream ticket:

I'm not sure if anything can be done here, but currently ns-slapd will crash/exit if /var fills up.  Now the database is mainly read-only so it seems that the only thing that would be written to would be the access log.  I would rather that the server stay active and loose access log entries than have the server go down.
Comment 1 RHEL Product and Program Management 2012-07-10 02:18:18 EDT
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.
Comment 2 RHEL Product and Program Management 2012-07-10 18:57:33 EDT
This request was erroneously removed from consideration in Red Hat Enterprise Linux 6.4, which is currently under development.  This request will be evaluated for inclusion in Red Hat Enterprise Linux 6.4.
Comment 4 Sankar Ramalingam 2013-01-24 07:00:07 EST
Request you to please add verification steps.
Comment 5 mreynolds 2013-01-24 09:24:04 EST
Here is the design doc:


Note, this is a new feature, not a bug fix.

To do a basic test set this in cn=config

nsslapd-disk-monitioring: on
nsslapd-disk-monitoring-threshold: ????  <value in bytes>
nsslapd-disk-monitoring-grace-period: 1 <one minute>
nsslapd-disk-monitoring-logging_critical: on

The idea is that if we get too low on disk space the DS will shutdown itself down gracefully.

What triggers the shutdown is when we get to less than half of threshold, and we exceed the grace period.

So if the available disk space is 8 gigs, and we set the threshold to 4 gigs, then when the available disk space gets below 2 gigs we goto into "shutdown mode".  The server will wait for the length of the grace period (in this example it's 1 minute), and if the available disk space is still less than half of the threshold we shutdown the server.

As for the treshold value, you need to know how much availalble space is on the test machine, and then set a value thats sufficient enough to trigger the issue.

To be honest there are many different scenarios to test, but this will test the basic functionality.
Comment 6 mreynolds 2013-01-24 10:29:54 EST
I forgot to mention, you need to restart the DS after setting the config attributes.
Comment 7 Ján Rusnačko 2013-01-30 05:38:33 EST
[jrusnack@dstet ~]$ ldapsearch -h localhost -p 22222 -D "cn=directory manager " -w Secret123 -LLL -b "cn=config" | grep disk-monitoring
nsslapd-disk-monitoring: on
nsslapd-disk-monitoring-threshold: 2000000000
nsslapd-disk-monitoring-grace-period: 1
nsslapd-disk-monitoring-logging-critical: on
nsslapd-disk-monitoring-preserve-logging: off

[jrusnack@dstet ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
                       13G   10G  1.8G  86% /
tmpfs                 499M  4.0K  499M   1% /dev/shm
/dev/vda1             485M   77M  383M  17% /boot

[jrusnack@dstet ~]$ tail /var/log/dirsrv/slapd-dstet/errors
[30/Jan/2013:05:28:28 -0500] - Disk space is too low on disk (/), remaining space: 1798232 Kb, disabling access and audit logging

Fill up the disk space...

[jrusnack@dstet ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
                       13G   12G  444M  97% /
tmpfs                 499M  4.0K  499M   1% /dev/shm
/dev/vda1             485M   77M  383M  17% /boot
[jrusnack@dstet ~]$ tail /var/log/dirsrv/slapd-dstet/errors
[30/Jan/2013:05:34:02 -0500] - Disk space is now within acceptable levels.
[30/Jan/2013:05:35:16 -0500] - Disk space is too low on disk (/), remaining space: 976316 Kb, disabling access and audit logging.
[30/Jan/2013:05:35:27 -0500] - Disk space is too low on disk (/), remaining space: 454208 Kb, deleting rotated logs.
[30/Jan/2013:05:35:37 -0500] - Disk space on (/) is too far below the threshold(2000000000 bytes).  Waiting 1 minutes for disk space to be cleaned up before shutting slapd down...
[30/Jan/2013:05:36:37 -0500] - Disk space is still too low (454208 Kb).  Signaling slapd for shutdown...
[30/Jan/2013:05:36:37 -0500] - slapd shutting down - signaling operation threads
[30/Jan/2013:05:36:37 -0500] - slapd shutting down - closing down internal subsystems and plugins
[30/Jan/2013:05:36:37 -0500] - Waiting for 4 database threads to stop
[30/Jan/2013:05:36:37 -0500] - All database threads now stopped
[30/Jan/2013:05:36:37 -0500] - slapd stopped.

Comment 9 errata-xmlrpc 2013-02-21 03:17:21 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.