Bug 986857

Summary: Disk Monitoring not checking filesystem with logs
Product: Red Hat Enterprise Linux 6 Reporter: Ján Rusnačko <jrusnack>
Component: 389-ds-baseAssignee: Rich Megginson <rmeggins>
Status: CLOSED ERRATA QA Contact: Sankar Ramalingam <sramling>
Severity: high Docs Contact:
Priority: urgent    
Version: 6.4CC: jgalipea, jwest, mreynolds, nhosoi, nkinder
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 389-ds-base-1.2.11.15-22.el6 Doc Type: Bug Fix
Doc Text:
Cause: Improper processing of the feature's configuration settings (logging critical is off) Consequence: If logging is not set to critical, then logs directory is incorrectly not checked for disk space. Fix: Fix the logic when processing the configuration settings Result: The log directories are checked regardless if logging is to set to critical or not.
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-11-21 21:11:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 987850    

Description Ján Rusnačko 2013-07-22 09:15:44 UTC
Description of problem:
When nsslapd-disk-monitoring-logging-critical is set to "on", disk monitoring plugin monitors filesystem containing DS logs. If filesystem is running out of space, it triggers disk monitoring plugin.

However, when nsslapd-disk-monitoring-logging-critical is set to "off", disk monitoring plugin is not checking mentioned filesystem. So, in case when there is not enough space to store logs, disk monitoring is not triggered, even though deleting rotated logfiles would help.

Version-Release number of selected component (if applicable):
389-ds-base-1.2.11.15-17.el6_4.x86_64

How reproducible:
always

Steps to Reproduce:
1. Create a separate filesystem and mount at /var/log/dirsrv/slapd-dstet, configure disk monitoring plugin:

[jrusnack@dstet dstet]$ ldapsearch -D "cn=directory manager" -w Secret123 -b "cn=config" -s base | grep "nsslapd-disk-monitoring"
nsslapd-disk-monitoring: on
nsslapd-disk-monitoring-threshold: 30000000
nsslapd-disk-monitoring-grace-period: 1
nsslapd-disk-monitoring-logging-critical: off
[jrusnack@dstet dstet]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_dstet-lv_root
                      9.4G  7.3G  1.8G  81% /
tmpfs                 499M   76K  499M   1% /dev/shm
/dev/vda1             485M   33M  427M   8% /boot
/home/jrusnack/testfs
                       39M  4.5M   33M  13% /var/log/dirsrv/slapd-dstet
[jrusnack@dstet dstet]$ dd if=/dev/zero of=/var/log/dirsrv/slapd-dstet
slapd-dstet/     slapd-dstet.bak/ 
[jrusnack@dstet dstet]$ dd if=/dev/zero of=/var/log/dirsrv/slapd-dstet/foo bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0435152 s, 241 MB/s
[jrusnack@dstet dstet]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_dstet-lv_root
                      9.4G  7.3G  1.8G  81% /
tmpfs                 499M   76K  499M   1% /dev/shm
/dev/vda1             485M   33M  427M   8% /boot
/home/jrusnack/testfs
                       39M   15M   23M  40% /var/log/dirsrv/slapd-dstet

# filesystem containing logs is now filled below the threshold, but disk monitoring plugin is not triggered and no error messages present in the error log

2. Try filling / below the threshold:

[jrusnack@dstet dstet]$ dd if=/dev/zero of=/home/jrusnack/foo bs=1K count=1768160
1768160+0 records in
1768160+0 records out
1810595840 bytes (1.8 GB) copied, 46.3845 s, 39.0 MB/s
[jrusnack@dstet dstet]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_dstet-lv_root
                      9.4G  8.9G   23M 100% /
tmpfs                 499M   76K  499M   1% /dev/shm
/dev/vda1             485M   33M  427M   8% /boot
/home/jrusnack/testfs
                       39M   15M   23M  40% /var/log/dirsrv/slapd-dstet
[jrusnack@dstet dstet]$ sudo cat /var/log/dirsrv/slapd-dstet/errors | grep "is too low"
[22/Jul/2013:11:09:28 +0200] - Disk space is too low on disk (/), remaining space: 22996 Kb, disabling access and audit logging.
[22/Jul/2013:11:09:38 +0200] - Disk space is too low on disk (/), remaining space: 22996 Kb, deleting rotated logs.
[22/Jul/2013:11:09:58 +0200] - Disk space is too low on disk (/), remaining space: 22992 Kb

Actual results:
Disk monitoring plugin is not monitoring available space of /var/log/dirsrv/<instance name> mountpoint when nsslapd-disk-monitoring-logging-critical is set to "off".

Additional info:
See related bugs
Bug 982621 - Disk monitoring not shutting down with nsslapd-disk-monitoring-logging-critical set to off 
Bug 972930 - RHDS not shutting down when disk monitoring threshold is reached to half.

Comment 2 mreynolds 2013-07-22 15:49:30 UTC
This behaviour was by design, but it should be changed.

Comment 3 Nathan Kinder 2013-07-22 19:12:33 UTC
Upstream ticket:
https://fedorahosted.org/389/ticket/47441

Comment 7 Ján Rusnačko 2013-08-23 11:38:48 UTC
Tests are passing, verified on 389-ds-base-1.2.11.15-22.el6.x86_64

Comment 8 errata-xmlrpc 2013-11-21 21:11:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1653.html