Bug 1202363 - After activating iscsi domain, can't add a new disk due to lack of space though there should be space.
Summary: After activating iscsi domain, can't add a new disk due to lack of space thou...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.5.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ovirt-3.6.0-rc
: 3.6.0
Assignee: Vered Volansky
QA Contact: Ori Gofen
URL:
Whiteboard:
Depends On:
Blocks: 1211057
TreeView+ depends on / blocked
 
Reported: 2015-03-16 13:27 UTC by Raz Tamir
Modified: 2016-05-26 01:51 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, when creating or activating a storage domain, there was a small window (a few seconds) that the used and available sizes were initialized with the value 0 where it should have been a null value and caused errors like 'Low disk space on Storage Domain'. With this update, the used and available sizes are set to null during the initialization window as it should have and no low disk errors are returned.
Clone Of:
: 1211057 (view as bug list)
Environment:
Last Closed: 2015-03-17 15:07:18 UTC
oVirt Team: Storage
Target Upstream Version:
Embargoed:
amureini: needinfo-


Attachments (Terms of Use)
engine log (9.81 MB, text/plain)
2015-03-16 13:27 UTC, Raz Tamir
no flags Details
vdsm.log (1.07 MB, text/plain)
2015-03-17 14:28 UTC, Carlos Mestre González
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 39004 0 None None None Never
oVirt gerrit 39028 0 master MERGED core: initializing domain sizes with null Never

Description Raz Tamir 2015-03-16 13:27:23 UTC
Created attachment 1002274 [details]
engine log

Description of problem:
When activating a storage domain, for a few seconds it's size and available size are not available and that's considerd as "Low disk space on Storage Domain"




Version-Release number of selected component (if applicable):


How reproducible:



Steps to Reproduce:
1. activate storage domain
2. look at it's size and available size
3.

Actual results:
size and available size are [N/A]


Expected results:


Additional info:
In the engine.log attached search for "2015-03-14 02:46:13"

Comment 1 Carlos Mestre González 2015-03-16 14:42:09 UTC
Regarding the actual results:
Rest API doesn't return available/used properties (only committed) for the storage domain.


This is an issue since in automation because the scenario of activating the storage domain and immediately after the this storage domain is 'Active', create a vm/disk, this action will fail with:

-> Cannot add VM. Low disk space on Storage Domain iscsi_2.

even though the domain is active.

Comment 2 Carlos Mestre González 2015-03-17 14:20:16 UTC
How reproducible:
10% ?

This happens to certain luns and not others (from the same storage server group), but I don't see any difference. This is a setup that has being up running for weeks different tests amount of tests, I'm not sure is that has to do with anything (some corruption?).

Comment 3 Carlos Mestre González 2015-03-17 14:28:34 UTC
Created attachment 1002828 [details]
vdsm.log

This setup contains 3 nfs domains and 3 iscsi domains (nfs_X and iscsi_x).
From 2015-03-17 15:51:37 the test starts, deactivating the master domain and activating again (iscsi_2).

For ~ ten seconds afer the domains is 'active' the available size and used size properties in rest API (N/A Webui) are missing (15:52:27 -> 15:52:38)

Comment 4 Allon Mureinik 2015-03-17 15:07:18 UTC
There are three separate issues here:

1. The domain's size being reported as unknown (null) until domain monitoring is run. This issue has been in place since RHEVM 3.0, at least, and frankly, it's not worth the effort to fix. It would require an extensive overhauling to the domain monitoring, for a really small gain, as the issue self corrects in a few minutes.

2. The fact you cannot create a disk until the size is updated. This issue has already been addressed in oVirt 3.5.2 (see bug 1177147), which RHEV 3.5.1 was rebased on. According to the report, this bug reproduced in 3.5.0, which is expected. If it also reproduces in the latest build of 3.5.1, please set bug 1177147 as Failed QA.

3. The fact that a warning is issues while the domain's size is not updated. This too was handled in bug 1177147.

I'm closing this bug as a duplicate.

*** This bug has been marked as a duplicate of bug 1177147 ***

Comment 5 Vered Volansky 2015-03-19 09:53:21 UTC
(In reply to Allon Mureinik from comment #4)
> There are three separate issues here:
> 
> 1. The domain's size being reported as unknown (null) until domain
> monitoring is run. This issue has been in place since RHEVM 3.0, at least,
> and frankly, it's not worth the effort to fix. It would require an extensive
> overhauling to the domain monitoring, for a really small gain, as the issue
> self corrects in a few minutes.
> 
> 2. The fact you cannot create a disk until the size is updated. This issue
> has already been addressed in oVirt 3.5.2 (see bug 1177147), which RHEV
> 3.5.1 was rebased on. According to the report, this bug reproduced in 3.5.0,
> which is expected. If it also reproduces in the latest build of 3.5.1,
> please set bug 1177147 as Failed QA.
> 
> 3. The fact that a warning is issues while the domain's size is not updated.
> This too was handled in bug 1177147.
> 
> I'm closing this bug as a duplicate.
> 
> *** This bug has been marked as a duplicate of bug 1177147 ***

bz1177147 fixed the problematic flow in creating a new domain. This looks like a different flow concerning activating a domain, reopening in order to handle issue no.2 .

Comment 6 Vered Volansky 2015-03-19 14:38:20 UTC
Carlos, regarding bz1177147#8, does this mean this flow was verified without the patch as well?

Comment 7 Carlos Mestre González 2015-03-19 16:00:52 UTC
This issue is still happening in vt14.

Comment 8 Vered Volansky 2015-03-23 07:23:34 UTC
Carlos,

We have no issue with not showing available size and used size properties in rest API. This is the desired behaviour in this case. Not be able to add disks in this scenario is indeed an issue.
Our fix to this will cause the above properties not be displayed in more scenarios (all concerning connectivity to domains).
Would you like to run your automation test with the patch before we're merging it?

Comment 9 Carlos Mestre González 2015-03-23 11:29:11 UTC
The issue indeed is not being able to create disks even though the storage domain is active.

Could you clarify what scenarios are those when it will not be displayed?

We're not going to have much problems in automation test, only for those test that indeed check those values, but we can always wait for the engine to report them.

Comment 10 Vered Volansky 2015-03-23 12:40:45 UTC
(In reply to Carlos Mestre González from comment #9)

> Could you clarify what scenarios are those when it will not be displayed?
1. Right after creating a new storage domain, for several seconds until the values are polled.
2. After recovery from connectivity issues or activation of a domain.

Comment 11 Vered Volansky 2015-03-23 12:42:44 UTC
The patch that aims to fix this issue is built and is closely coupled with two other patches, touching the same areas, and is not complete without them. Adding them to external trackers.

Comment 13 Allon Mureinik 2015-04-13 12:07:36 UTC
Vered, can you add the doctext for this bz please?

Comment 14 Allon Mureinik 2015-04-14 11:33:50 UTC
Copied doctext from the zstream clone (bug #1211057).

Comment 15 Ori Gofen 2015-06-29 13:25:35 UTC
Verified on 3.6.0.3

Comment 16 Allon Mureinik 2016-03-10 10:38:23 UTC
RHEV 3.6.0 has been released, setting status to CLOSED CURRENTRELEASE

Comment 17 Allon Mureinik 2016-03-10 10:38:34 UTC
RHEV 3.6.0 has been released, setting status to CLOSED CURRENTRELEASE

Comment 18 Allon Mureinik 2016-03-10 10:44:25 UTC
RHEV 3.6.0 has been released, setting status to CLOSED CURRENTRELEASE

Comment 19 Allon Mureinik 2016-03-10 12:01:07 UTC
RHEV 3.6.0 has been released, setting status to CLOSED CURRENTRELEASE


Note You need to log in before you can comment on or make changes to this bug.