Description of problem: Warning message of low space is not received when a Storage domain containing low space is imported to a RHV environment. Version-Release number of selected component (if applicable): How reproducible: 100 % Steps to Reproduce: 1. Add disks to existing Storage domain to bring down the free space below the threshold configured in "Warning Low Space Indicator". Warning message may be displayed at this point. 2. Remove the Storage domain & Import it back. Actual results: No warning messages is received after importing the storage domain with low space. Expected results: Warning message should be received after importing the storage domain with low space. Additional info: ovirt-engine restart on rhvm server triggers the warning message on the imported storage domain with low space.
After testing a few different scenarios I have found that sometimes the Warning message is correctly shows in the Events. e.g.: 1. On Freshly started engine, try to import and activate Storage domain with low space. Warning message is displayed! The Warning Message is correctly created after the warning condition on the Storage Domain is met. But Unfortunately the message is not displayed if there was similar message displayed in the past 12 hours. The message is rejected by the EventFloodRegulator used in AuditLog (current configuration is set to 12 hours for this specific warning). I think one of the problems is that the warning is created without any ID assigned (storage domain ID, or pool ID, etc.), so it is a generic IRS_DISK_SPACE_LOW Warning, and when one of this warnings is shown anywhere in the system for any Storage domain, it will not be shown for this and all the other Storage domains for next 12 hours. That is why in the scenario presented in the problem description, after adding Disk to storage domain such WARNING is shown, and then when importing Storage domain with LOW SPACE it will not be shown. Adding Storage domain ID to the WARNING will change the behavior, so that the message is shown one in 12 hours for each Storage Domain. So when importing domain that was not there in the last 12 hours, it will show WARNING, when creating a new domain it will show WARNING, and when adding Disk so the free space of the domain becomes too low, and there was no WARNING in last 12 hours for THIS domain - it will show WARNING. But, it will not show WARNING if the space is again missing on a domain on which the message was already shown in last 12 hours. So if we also want to change this behavior for WARNINGS to be always shown after action is executed, we need to find another way. BTW: there is a similar case with "Critically low disk space" (time 15 minutes, but also without ID so works globally).
I tried to import storage domain with low space but I didn't see any warnings. Please prove a flow to verify this fix. ovirt-engine-4.4.0-0.33.master.el8ev.noarch
Previously one warning on one storage domain was actually silencing the whole system - no other warnings were show - this was fixed. But I need to repeat one important detail: If one removes and imports the same storage domain again within configured time window (12 h) the warning will not be shown again if it was shown in last 12 hours.
(In reply to Bartosz Rybacki from comment #4) > Previously one warning on one storage domain was actually silencing the > whole system - no other warnings were show - this was fixed. > > > But I need to repeat one important detail: > > If one removes and imports the same storage domain again within configured > time window (12 h) the warning will not be shown again if it was shown in > last 12 hours. Is this ok for you?
Hi Evelina, Are you able to test it according to comment#2? If the basic functionality works, we probably can expect the side corner. Ideally, we should probably reset this mechanism on detach of a storage domain. But this may qualify for another bug/rfe. Feel free to file it. Thanks, Marina.
(In reply to Marina Kalinin from comment #6) > Hi Evelina, > > Are you able to test it according to comment#2? > If the basic functionality works, we probably can expect the side corner. > > Ideally, we should probably reset this mechanism on detach of a storage > domain. But this may qualify for another bug/rfe. Feel free to file it. > > Thanks, > Marina. Warning low space has default time window of 12 hours (so it can show once per 12 hours). According to the fix, now messages for distinct domain Ids can show. That means that if sd_1 has low space and sd_2 has low space - now we will see 2 warnings - one for each storage domain. But, this doesn't fix this bug - Warning message for low space is still not received when importing storage domain with low space in this window time (12h).
So, can we say that by Bartozs' fix we have actually fixed the root cause of the message not being deployed at all? And the title of hte bug is the side effect of this actual problem?
(In reply to Marina Kalinin from comment #8) > So, can we say that by Bartozs' fix we have actually fixed the root cause of > the message not being deployed at all? Not exactly, we discussed this with Bartosz, and according to Comment#2 the message is deployed but once in 12H per all storage domains, see details below. Before this fix: On a Freshly started engine, import& activate a specific SD 'A' with low space=> low space Warning message is displayed once for the next 12 hour for ALL storage domains. What did not work: After one of these warnings is shown anywhere in the system for any Storage domain, it will not be shown for this and all the other Storage domains for next 12 hours. Meaning that for the next 12 hours: - if SD 'A' detach/remove than import => No new Warning message is displayed. - if another SD 'B'/'C'/.... gets to low space => No new Warning message is displayed. After fix (add storage ID): The warning message is shown one in 12 hours for each Storage Domain. Meaning that for the next 12 hours: - If another SD 'B'/'C'/.... gets to low space => A new Warning message is displayed => this is fixed but this is not the specific customer scenario. - If SD 'A' detach/remove than import => No new Warning message is displayed, Original customer issue IS NOT FIXED. To clarify: This fix DOES NOT fix this specific customer scenario which is: SD 'A' detach/remove than import => customer expects a new(2nd) Warning message to be displayed. Verifying this bug means the specific customer scenario was fixed. In my eyes(QE) and after discussing this with Bartosz this fix does not resolve the specific customer scenario meaning this bug should be reopened unless there are other customer scenarios which is not described in this bug. Q for the customer/GSS: - Does the current fix sufficient for the customer? - Are there other customer scenarios (not mentioned in this bug) that this current fix does provide a solution for? Q for DEV: - How much effort will it take to fix this specific scenario? can DEV make it happen?
Please fix the originally reported issue. Thank you.
If the original problem isn't fixed so we should fix it properly. It is great that we improved the system behavior but in order to close this bug, we should fix it according to the customer's expected result.
The issue was fixed, now when performing the following steps a low disk space warning should appear: 1. Add disks to the existing Storage domain to bring down the free space below the threshold configured in "Warning Low Space Indicator". A warning message should be displayed at this point for the first time. 2. Remove the Storage domain and Import it back, at this point another warning message should be displayed even if less then 12 hours were passed.
This bug is in modified state and targeting 4.4.2. Can this be re-targeted to 4.4.1?
Verified on engine-4.4.1.2-0.10.el8ev
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: RHV Manager (ovirt-engine) 4.4 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:3247