Red Hat Bugzilla – Bug 1467767
Engine config CriticalSpaceActionBlocker doesn't terminate existing jobs/tasks
Last modified: 2018-03-09 17:51:35 EST
Description of problem:
The engine config CriticalSpaceActionBlocker at the default value of 5G is suppose to stall all operations if the storage space is less than 5GB . However if a concurrent series of operations are taking place , then the jobs/tasks continue without getting terminated and fills up the storage space causing the snapshot operations if involved to be in illegal state .
Version-Release number of selected component (if applicable):
This can be easily reproducible by creating a pool of VM"s or using a API script to create a snapshot for all the VM"s on RHEV causing the storage to fill up and go beyond the CriticalSpaceActionBlocker specified value.
Steps to Reproduce:
1. Configure CriticalSpaceActionBlocker to be at 5G on the engine server. (default is 5GB).
2. Choose a storage SD low on free space eg. 20G or less
3.1 Create a pool of VM's with each 5GB disk that would result in the Storage space going less than 5G and you should see the error as follows
3.2 Create a snapshot for all VM's on the RHEV environment simultaneously using a API script in place.
Resulted in the below error.
2017-07-05 00:24:37,209-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7)  EVENT_ID: IRS_DISK_SPACE_LOW_ERROR(201), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. master-iscsi-SD domain has 2 GB of free space.
* Storage goes to 0 bytes free
* If snapshot's are involved , then it ends up in illegal state.
* Pre-calculate the size required for the operations.
* Terminate operations once the threshold is reached.
There are several things going on here.
I'll start from the end - even without any management operations (such as creating snapshots), your storage can still run out by running thinly-provisioned VMs that need to grow over time.
Moreover, if the storage **server** is thinly provisioning RHV's storage, it's essentially "lying" to RHV, and you can still run out of storage space even if RHV is doing everything correctly.
Canceling ongoing tasks is a mechanism we don't have (regardless of this bug, even as an admin action in REST/GUI), and frankly, I don't see this happening anytime soon, if at all - just to set the expectations.
Regardless of the above, RHV's side could, in theory, be improved to take into consideration "in flight" operations, but that would complicate the solution significantly, and we need to think very carefully if
1. The investment of effort is worth it
2. This is indeed a zstream item
An easier relief plan could be to suggest an improvement to the customer's script that checks the available space by itself before creating all those snapshots.
Ribu, would that help relieve the immediate escalation?
Yes I understand your point. Additionally this can be reproduced even if we create a pool of VM's simultaneously with the space running out all of a sudden and hence it need not be a script or API call for this purpose.
Yaniv, please review comment 4 with my analysis on the problem.
If we do want to support such "inflight" operation checking, it's too much work for a zstream.
We could try to push it for 4.2, but it's a stretch even for that.
I suggest treating this as an RFE, and targetting against other RFEs.
(In reply to Allon Mureinik from comment #7)
> Yaniv, please review comment 4 with my analysis on the problem.
> If we do want to support such "inflight" operation checking, it's too much
> work for a zstream.
> We could try to push it for 4.2, but it's a stretch even for that.
> I suggest treating this as an RFE, and targetting against other RFEs.
I think we should wait to see how broad the issue is, for now pushing to 4.3.
(In reply to Yaniv Lavi from comment #8)
> I think we should wait to see how broad the issue is, for now pushing to 4.3.
For my customer, who's working near the limit of available storage, this bug is critical.
Especially the result is ending up in illegal state images, this should be fixed as soon as possible. If not possible, there should be really some kind of workaround, to not run into this issue again. My customer is really sensitive regarding illegal/corrupted images or filesystems.
(In reply to Steffen Froemer from comment #9)
> (In reply to Yaniv Lavi from comment #8)
> > I think we should wait to see how broad the issue is, for now pushing to 4.3.
> For my customer, who's working near the limit of available storage, this bug
> is critical.
> Especially the result is ending up in illegal state images, this should be
> fixed as soon as possible. If not possible, there should be really some kind
> of workaround, to not run into this issue again. My customer is really
> sensitive regarding illegal/corrupted images or filesystems.
Allon, looks like an image corruption issue here, can you have a second look on this one based on comment #8
(In reply to Moran Goldboim from comment #10)
> (In reply to Steffen Froemer from comment #9)
> > (In reply to Yaniv Lavi from comment #8)
> > >
> > > I think we should wait to see how broad the issue is, for now pushing to 4.3.
> > For my customer, who's working near the limit of available storage, this bug
> > is critical.
> > Especially the result is ending up in illegal state images, this should be
> > fixed as soon as possible. If not possible, there should be really some kind
> > of workaround, to not run into this issue again. My customer is really
> > sensitive regarding illegal/corrupted images or filesystems.
> Allon, looks like an image corruption issue here, can you have a second look
> on this one based on comment #8
There's no corruption here - just a bunch of images that failed creation and are now "half-baked" and unusable to RHV. No data is compromised.
1. In order to prevent something like this from ever happening - very doable, see my suggestion in comment 5.
2. To recover from it - just delete the half-baked images.
Regarding providing the feature - as noted in comment 4, this is doable (with some limitations/caveats), but is a non-trivial effort. If PM stakeholders think this should be prioritized, we can discuss if/when we can attempt to provide this.
This is also an issue if you move many disks at once from one storage domain to another. While ovirt checks the free space for an individual image is available it does not check the combined sizes.
Thinking about this again and I understand it can be a challenging task to check the available storage at CanDoAction of Snapshot creation, since we don't know about tasks trying to perform storage-heavy operation.
But then maybe we can add a warning to the user that their storage reached the critical threshold and the operation may fail, if the storage capacity is not increased. This should at least notify the user about the potential failure and leave them at their own risk. But hopefully this will trigger a call to the storage team asking to expand the storage (or similar action).
^about tasks^about all the tasks
Since the issue is ongoing and being faced by customers from time to time as indicated on my comment #0 it would be worth adding a task if possible to pre-calculate the disk size of the storage domain before proceeding.
Please also check bz#1343592.
Quoting my reply on the mailing list for posterity:
This request is complicated, to say the least, in several levels.
First, in the most basic level - RHV currently has no notion of tracking "in flight" storage operations. We could attempt to add something like that (in a analogous way to the mechanism used for quota calculations), but that's a *very* big and risky change. It's probably doable, but I sincerely doubt that given the cost it would be the best use of resources (although PM are welcome to weigh in here, of course!).
Second, even if we did that, we'd still a lying to our users. E.g., if you have 10GB you can schedule a 5GB series of operations. But even if this scheduling takes into account the series of events, there's nothing preventing a heavily loaded VM the uses a thinly provisioned disk to gobble up the free space and you'd be back to square one.
Third, even if you don't use thinly provisioned VMs (which the customer probably is, since this entire discussion has been around snapshots and pools), anything RHV does will be fragile (i.e., broken, if you ignore being PC for a moment), since the underlying storage would almost definitely be thinly provisioned from the array's side, so any data RHV has about free space is just fake news.
Operatively, looking forward:
1. The default when taking a live snapshot is to also take a memory snapshot. This is useful in some situations, but if you're only taking a snapshot for backup purposes, it's pointless. These snapshots consume quite a bit of storage (roughly the size of the VM's memory, plus the virtual screen(s) buffer, plus some overhead), so make sure you take your snapshots *without* memory
2. Once 4.2 is out, make sure to upgrade to it. It enhances live-merge by optimizing the size of the merged snapshot once the merge is done, and prevents the long tail of slowly growing snapshots (https://bugzilla.redhat.com/show_bug.cgi?id=1168327)
3. Batching snapshots is a good idea, but I agree with Yaniv that one at a time is a bit too aggressive. Choose a reasonable number (let's say 10), and edit the storage domain's "Critical Space Action Blocker (GB)" to be that number of GBs, +1 for good measure - that should ensure you have enough space to create these snapshots, at least from RHV's side (the restrictions on thin provisioning still apply, of course).
Should we close this bug WONTFIX based on the last explanation from Allon?