Bug 830331 - ns-slapd exits/crashes if /var fills up
Summary: ns-slapd exits/crashes if /var fills up
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: 389-ds-base
Version: 6.4
Hardware: Unspecified
OS: Unspecified
medium
unspecified
Target Milestone: rc
: ---
Assignee: Rich Megginson
QA Contact: Sankar Ramalingam
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-06-08 21:06 UTC by Nathan Kinder
Modified: 2013-07-23 11:24 UTC (History)
3 users (show)

Fixed In Version: 389-ds-base-1.2.11.12-1.el6
Doc Type: Enhancement
Doc Text:
Feature: Disk Monitoring Reason: If certain disk partitions fill up, the server should take action to free up space, and/or shutdown cleanly. Result (if any): DS does not crash when disk fills up.
Clone Of:
Environment:
Last Closed: 2013-02-21 08:17:21 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2013:0503 normal SHIPPED_LIVE Moderate: 389-ds-base security, bug fix, and enhancement update 2013-02-21 08:18:44 UTC

Description Nathan Kinder 2012-06-08 21:06:48 UTC
This bug is created as a clone of upstream ticket:
https://fedorahosted.org/389/ticket/315

I'm not sure if anything can be done here, but currently ns-slapd will crash/exit if /var fills up.  Now the database is mainly read-only so it seems that the only thing that would be written to would be the access log.  I would rather that the server stay active and loose access log entries than have the server go down.

Comment 1 RHEL Product and Program Management 2012-07-10 06:18:18 UTC
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.

Comment 2 RHEL Product and Program Management 2012-07-10 22:57:33 UTC
This request was erroneously removed from consideration in Red Hat Enterprise Linux 6.4, which is currently under development.  This request will be evaluated for inclusion in Red Hat Enterprise Linux 6.4.

Comment 4 Sankar Ramalingam 2013-01-24 12:00:07 UTC
Request you to please add verification steps.

Comment 5 mreynolds 2013-01-24 14:24:04 UTC
Here is the design doc:

http://port389.org/wiki/Disk_Monitoring

Note, this is a new feature, not a bug fix.

To do a basic test set this in cn=config

nsslapd-disk-monitioring: on
nsslapd-disk-monitoring-threshold: ????  <value in bytes>
nsslapd-disk-monitoring-grace-period: 1 <one minute>
nsslapd-disk-monitoring-logging_critical: on

The idea is that if we get too low on disk space the DS will shutdown itself down gracefully.

What triggers the shutdown is when we get to less than half of threshold, and we exceed the grace period.

So if the available disk space is 8 gigs, and we set the threshold to 4 gigs, then when the available disk space gets below 2 gigs we goto into "shutdown mode".  The server will wait for the length of the grace period (in this example it's 1 minute), and if the available disk space is still less than half of the threshold we shutdown the server.

As for the treshold value, you need to know how much availalble space is on the test machine, and then set a value thats sufficient enough to trigger the issue.

To be honest there are many different scenarios to test, but this will test the basic functionality.

Comment 6 mreynolds 2013-01-24 15:29:54 UTC
I forgot to mention, you need to restart the DS after setting the config attributes.

Comment 7 Ján Rusnačko 2013-01-30 10:38:33 UTC
[jrusnack@dstet ~]$ ldapsearch -h localhost -p 22222 -D "cn=directory manager " -w Secret123 -LLL -b "cn=config" | grep disk-monitoring
nsslapd-disk-monitoring: on
nsslapd-disk-monitoring-threshold: 2000000000
nsslapd-disk-monitoring-grace-period: 1
nsslapd-disk-monitoring-logging-critical: on
nsslapd-disk-monitoring-preserve-logging: off

[jrusnack@dstet ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_dstet-lv_root
                       13G   10G  1.8G  86% /
tmpfs                 499M  4.0K  499M   1% /dev/shm
/dev/vda1             485M   77M  383M  17% /boot

[jrusnack@dstet ~]$ tail /var/log/dirsrv/slapd-dstet/errors
...
[30/Jan/2013:05:28:28 -0500] - Disk space is too low on disk (/), remaining space: 1798232 Kb, disabling access and audit logging
...

Fill up the disk space...

[jrusnack@dstet ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_dstet-lv_root
                       13G   12G  444M  97% /
tmpfs                 499M  4.0K  499M   1% /dev/shm
/dev/vda1             485M   77M  383M  17% /boot
[jrusnack@dstet ~]$ tail /var/log/dirsrv/slapd-dstet/errors
[30/Jan/2013:05:34:02 -0500] - Disk space is now within acceptable levels.
[30/Jan/2013:05:35:16 -0500] - Disk space is too low on disk (/), remaining space: 976316 Kb, disabling access and audit logging.
[30/Jan/2013:05:35:27 -0500] - Disk space is too low on disk (/), remaining space: 454208 Kb, deleting rotated logs.
[30/Jan/2013:05:35:37 -0500] - Disk space on (/) is too far below the threshold(2000000000 bytes).  Waiting 1 minutes for disk space to be cleaned up before shutting slapd down...
[30/Jan/2013:05:36:37 -0500] - Disk space is still too low (454208 Kb).  Signaling slapd for shutdown...
[30/Jan/2013:05:36:37 -0500] - slapd shutting down - signaling operation threads
[30/Jan/2013:05:36:37 -0500] - slapd shutting down - closing down internal subsystems and plugins
[30/Jan/2013:05:36:37 -0500] - Waiting for 4 database threads to stop
[30/Jan/2013:05:36:37 -0500] - All database threads now stopped
[30/Jan/2013:05:36:37 -0500] - slapd stopped.

Verified.

Comment 9 errata-xmlrpc 2013-02-21 08:17:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0503.html


Note You need to log in before you can comment on or make changes to this bug.