Bug 982621
Summary: | Disk monitoring not shutting down with nsslapd-disk-monitoring-logging-critical set to off | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 6 | Reporter: | Ján Rusnačko <jrusnack> |
Component: | 389-ds-base | Assignee: | mreynolds |
Status: | CLOSED DUPLICATE | QA Contact: | Sankar Ramalingam <sramling> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | 6.4 | CC: | jgalipea, jrusnack, nkinder |
Target Milestone: | rc | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2013-07-12 21:25:55 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Ján Rusnačko
2013-07-09 12:24:02 UTC
The server is not going into shut-down mode because you are not continuing to lose disk space. See the design doc: http://directory.fedoraproject.org/wiki/Disk_Monitoring The feature only wants to shut-down the server as a last resort. If things remain stable, it doesn't do anything. I do think there is room for improvement in this area though. It looks like you just consumed a chunk of disk space at one shot. If you continue to consume disk space does it go into shutdown mode? (In reply to mreynolds from comment #4) > The server is not going into shut-down mode because you are not continuing > to lose disk space. See the design doc: > > http://directory.fedoraproject.org/wiki/Disk_Monitoring > > The feature only wants to shut-down the server as a last resort. If things > remain stable, it doesn't do anything. I do think there is room for > improvement in this area though. > > It looks like you just consumed a chunk of disk space at one shot. If you > continue to consume disk space does it go into shutdown mode? According to the design doc: "Once the available disk space on any of the disks gets below the threshold we start taking action." The space was consumed in one shot, however, since the threshold was passed, as the first thing verbose logging should get disabled, right ? It was not, therefore I assumed disk monitoring plugin was not even triggered when the space dropped below the threshold. I will investigate what will happen if available space continues to drop and report back. (In reply to Ján Rusnačko from comment #5) > (In reply to mreynolds from comment #4) > > The server is not going into shut-down mode because you are not continuing > > to lose disk space. See the design doc: > > > > http://directory.fedoraproject.org/wiki/Disk_Monitoring > > > > The feature only wants to shut-down the server as a last resort. If things > > remain stable, it doesn't do anything. I do think there is room for > > improvement in this area though. > > > > It looks like you just consumed a chunk of disk space at one shot. If you > > continue to consume disk space does it go into shutdown mode? > > According to the design doc: > > "Once the available disk space on any of the disks gets below the threshold > we start taking action." > > The space was consumed in one shot, however, since the threshold was passed, > as the first thing verbose logging should get disabled, right ? It was not, > therefore I assumed disk monitoring plugin was not even triggered when the > space dropped below the threshold. This is the area that can use some improvement. It expects that the threshold will be hit first, but not past the halfway mark. So it makes a complete pass, then on the next pass if space continues to drop it will go into shutdown mode. But in your case, even though you are past the threshold halfway mark, it will not enter the shutdown code because it happened in one shot and did not continue to lose disk space. So yes, there is a bug to fix, but I just wanted you to verify this behavior. Thanks, Mark > > I will investigate what will happen if available space continues to drop and > report back. (In reply to mreynolds from comment #6) > (In reply to Ján Rusnačko from comment #5) > > (In reply to mreynolds from comment #4) > > > The server is not going into shut-down mode because you are not continuing > > > to lose disk space. See the design doc: > > > > > > http://directory.fedoraproject.org/wiki/Disk_Monitoring > > > > > > The feature only wants to shut-down the server as a last resort. If things > > > remain stable, it doesn't do anything. I do think there is room for > > > improvement in this area though. > > > > > > It looks like you just consumed a chunk of disk space at one shot. If you > > > continue to consume disk space does it go into shutdown mode? > > > > According to the design doc: > > > > "Once the available disk space on any of the disks gets below the threshold > > we start taking action." > > > > The space was consumed in one shot, however, since the threshold was passed, > > as the first thing verbose logging should get disabled, right ? It was not, > > therefore I assumed disk monitoring plugin was not even triggered when the > > space dropped below the threshold. > > This is the area that can use some improvement. It expects that the > threshold will be hit first, but not past the halfway mark. So it makes a > complete pass, then on the next pass if space continues to drop it will go > into shutdown mode. But in your case, even though you are past the > threshold halfway mark, it will not enter the shutdown code because it > happened in one shot and did not continue to lose disk space. > > So yes, there is a bug to fix, but I just wanted you to verify this behavior. Ok, thanks ! > > Thanks, > Mark > > > > I will investigate what will happen if available space continues to drop and > > report back. I have gradually decreased amount of available space by 2MB every 2 seconds - disk monitoring plugin was not triggered. I was able to hit 0 free space without DS immediately shutting down. (In reply to Ján Rusnačko from comment #7) > (In reply to mreynolds from comment #6) > > (In reply to Ján Rusnačko from comment #5) > > > (In reply to mreynolds from comment #4) > > > > The server is not going into shut-down mode because you are not continuing > > > > to lose disk space. See the design doc: > > > > > > > > http://directory.fedoraproject.org/wiki/Disk_Monitoring > > > > > > > > The feature only wants to shut-down the server as a last resort. If things > > > > remain stable, it doesn't do anything. I do think there is room for > > > > improvement in this area though. > > > > > > > > It looks like you just consumed a chunk of disk space at one shot. If you > > > > continue to consume disk space does it go into shutdown mode? > > > > > > According to the design doc: > > > > > > "Once the available disk space on any of the disks gets below the threshold > > > we start taking action." > > > > > > The space was consumed in one shot, however, since the threshold was passed, > > > as the first thing verbose logging should get disabled, right ? It was not, > > > therefore I assumed disk monitoring plugin was not even triggered when the > > > space dropped below the threshold. > > > > This is the area that can use some improvement. It expects that the > > threshold will be hit first, but not past the halfway mark. So it makes a > > complete pass, then on the next pass if space continues to drop it will go > > into shutdown mode. But in your case, even though you are past the > > threshold halfway mark, it will not enter the shutdown code because it > > happened in one shot and did not continue to lose disk space. > > > > So yes, there is a bug to fix, but I just wanted you to verify this behavior. > Ok, thanks ! > > > > Thanks, > > Mark > > > > > > I will investigate what will happen if available space continues to drop and > > > report back. > I have gradually decreased amount of available space by 2MB every 2 seconds > - disk monitoring plugin was not triggered. I was able to hit 0 free space > without DS immediately shutting down. Yeah that's bad, and surprising. I'll start working on this right away. (In reply to mreynolds from comment #8) > (In reply to Ján Rusnačko from comment #7) > > (In reply to mreynolds from comment #6) > > > (In reply to Ján Rusnačko from comment #5) > > > > (In reply to mreynolds from comment #4) > > > > > The server is not going into shut-down mode because you are not continuing > > > > > to lose disk space. See the design doc: > > > > > > > > > > http://directory.fedoraproject.org/wiki/Disk_Monitoring > > > > > > > > > > The feature only wants to shut-down the server as a last resort. If things > > > > > remain stable, it doesn't do anything. I do think there is room for > > > > > improvement in this area though. > > > > > > > > > > It looks like you just consumed a chunk of disk space at one shot. If you > > > > > continue to consume disk space does it go into shutdown mode? > > > > > > > > According to the design doc: > > > > > > > > "Once the available disk space on any of the disks gets below the threshold > > > > we start taking action." > > > > > > > > The space was consumed in one shot, however, since the threshold was passed, > > > > as the first thing verbose logging should get disabled, right ? It was not, > > > > therefore I assumed disk monitoring plugin was not even triggered when the > > > > space dropped below the threshold. > > > > > > This is the area that can use some improvement. It expects that the > > > threshold will be hit first, but not past the halfway mark. So it makes a > > > complete pass, then on the next pass if space continues to drop it will go > > > into shutdown mode. But in your case, even though you are past the > > > threshold halfway mark, it will not enter the shutdown code because it > > > happened in one shot and did not continue to lose disk space. > > > > > > So yes, there is a bug to fix, but I just wanted you to verify this behavior. > > Ok, thanks ! > > > > > > Thanks, > > > Mark > > > > > > > > I will investigate what will happen if available space continues to drop and > > > > report back. > > I have gradually decreased amount of available space by 2MB every 2 seconds > > - disk monitoring plugin was not triggered. I was able to hit 0 free space > > without DS immediately shutting down. > > Yeah that's bad, and surprising. I'll start working on this right away. Oh, to be precise and avoid misunderstanding, I have first depleted space below half of the threshold in one shot, and only THEN decreased space by 2 MB till 0 free space. Jan, I have fixed all the reported issues. Should I do a respin of 1.2.11, or are you still running other tests? Thanks, Mark (In reply to mreynolds from comment #10) > Jan, I have fixed all the reported issues. Should I do a respin of 1.2.11, > or are you still running other tests? > > Thanks, > Mark Hi Mark, All the tests are automated and I have reported all issues I have found. Please continue with respin. Thank you, Jan Closing this as a duplicate of bug 972930. The fix for that bug was not correct, and caused the issue described in this report. We will continue to use 972930 for this issue, as it is already acked for the planned 6.4.z update. *** This bug has been marked as a duplicate of bug 972930 *** |