Bug 1178012 - RHEV: Faulty storage allocation checks when importing a disk from glance
Summary: RHEV: Faulty storage allocation checks when importing a disk from glance
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.4.0
Hardware: Unspecified
OS: All
medium
medium
Target Milestone: ovirt-3.6.0-rc
: 3.6.0
Assignee: Vered Volansky
QA Contact: Elad
URL:
Whiteboard:
Depends On: 1053750
Blocks: 960934
TreeView+ depends on / blocked
 
Reported: 2015-01-01 12:29 UTC by Allon Mureinik
Modified: 2016-03-10 12:06 UTC (History)
19 users (show)

Fixed In Version: ovirt-engine-3.5.0_vt6
Doc Type: Bug Fix
Doc Text:
Clone Of: 1053750
Environment:
Last Closed:
oVirt Team: Storage
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
engine log amd db dump (1.18 MB, application/x-gzip)
2015-01-13 10:59 UTC, Elad
no flags Details
vdsm log (717.02 KB, application/x-gzip)
2015-01-13 11:11 UTC, Elad
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 33844 0 None MERGED core: Import image from glance storage allocation 2020-03-09 13:17:05 UTC
oVirt gerrit 33850 0 None MERGED core: Import image from glance storage allocation 2020-03-09 13:17:04 UTC
oVirt gerrit 37505 0 master MERGED core: Fix import from glance storage allocation check 2020-03-09 13:17:09 UTC

Description Allon Mureinik 2015-01-01 12:29:32 UTC
+++ This bug was initially created as a clone of Bug #1053750 +++

+++ This bug was initially created as a clone of Bug #1053746 +++

+++ This bug was initially created as a clone of Bug #1053733 +++

+++ This bug was initially created as a clone of Bug #960934 +++

The storage requirements for the imported disk:

      | File Domain                             | Block Domain
 -----|-----------------------------------------|-------------
 qcow | preallocated : 1.1 * disk capacity      |1.1 * min(used ,capacity) 
      | sparse: 1.1 * min(used ,capacity)       |
 -----|-----------------------------------------|-------------
 raw  | preallocated: disk capacity             |disk capacity
      | sparse: min(used,capacity)

-------------------------------------------------------------------------------
This bug is a RHEV tracker for the QE team to verify against RHEVM 3.5.0

Comment 1 Allon Mureinik 2015-01-01 12:30:43 UTC
bug was put on ovirt-engine-config by mistake, please ignore the original assignment.

Comment 2 Elad 2015-01-12 10:03:58 UTC
The same as https://bugzilla.redhat.com/show_bug.cgi?id=1178010
Import an image from glance, while its size equales to the free space of the domain, is not blocked and the import operation is performed. After that, the domain is reported to have 0G free space.

Image size is 5368709120:


2015-01-12 11:53:40,402 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (org.ovirt.thread.pool-7-thread-31) [6cba6dbe] START, CreateImageVDSCommand( stor
agePoolId = 00000002-0002-0002-0002-0000000003e1, ignoreFailoverLimit = false, storageDomainId = 51ab53aa-9b94-4dec-b180-01a0c9de8915, imageGroupId = 93c411bb-699c-4987-8820-71a
5611f73ac, imageSizeInBytes = 5368709120, volumeFormat = RAW, newImageId = 4c0bd0f9-96af-4c6f-9f90-a6f62a32ae71, newImageDescription = vm-1_Disk1 (ae529be)), log id: 7f281ee9


After the import, the domain is reported with 0 GB free space:

2015-01-12 11:54:05,425 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [285829e8] Correlation ID: null, Call Sta
ck: null, Custom Event ID: -1, Message: Critical, Low disk space. fc1 domain has 0 GB of free space


Vered, the same as https://bugzilla.redhat.com/show_bug.cgi?id=1178010, I don't think it is the right behavior.

Comment 3 Vered Volansky 2015-01-12 15:57:34 UTC
(In reply to Elad from comment #2)
> The same as https://bugzilla.redhat.com/show_bug.cgi?id=1178010
> Import an image from glance, while its size equales to the free space of the
> domain, is not blocked and the import operation is performed. After that,
> the domain is reported to have 0G free space.

Equality is enough for us to do the operation, we're blocking at less than.
OGB is not a desirable status, but also not an illegal one, and is certainly not in the scope of this bz.

> Vered, the same as https://bugzilla.redhat.com/show_bug.cgi?id=1178010, I
> don't think it is the right behavior.
In that bug you described a less than situation where free_space < needed_space, so I don't think it's the same issue, please correct me if I'm wrong.

Comment 4 Elad 2015-01-13 10:56:07 UTC
I managed to import an image from Glance, which its size was bigger than the free space on the domain. Occurred on NFS domain. 
On iSCSI, the same action was blocked.


Free space on the domain was 6G and the image size is 7G  (imageSizeInBytes = 7516192768):

2015-01-13 12:06:47,806 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (org.ovirt.thread.pool-7-thread-12) [2384aa32] START, CreateImageVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000003e1, ignoreFailoverLimit = false, storageDomainId = 4b9b6b0a-31c7-4ed4-928b-771ef29e8c60, imageGroupId = 2bf52586-a3f1-4e33-9037-b121574aaab3, imageSizeInBytes = 7516192768, volumeFormat = RAW, newImageId = 122c26a9-7daf-4897-9105-0a8714b95c3c, newImageDescription = 1 (0aef12f)), log id: 1213fc86

During the import, the domain free space is decreasing until it runs out of space:

2015-01-13 12:07:06,620 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) [befb354] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Messag
e: Critical, Low disk space. lion1 domain has 5 GB of free space

The import operation failed:

2015-01-13 12:08:47,084 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-18) [541c8b14] Correlation ID: 573c8183, Call Stack: null, Custom Event ID: -1, Message: User admin failed to import image GlanceDisk-0aef12f to domain lion1.


The image gets removed after the failure:

2015-01-13 12:08:43,560 INFO  [org.ovirt.engine.core.bll.RemoveDiskCommand] (org.ovirt.thread.pool-7-thread-18) [19fff775] Running command: RemoveDiskCommand internal: true. Entities affected :  ID: 2bf52586-a3f1-4e33-9037-b121574aaab3 Type: DiskAction group DELETE_DISK with role type USER


I checked using vt13.6

Comment 5 Elad 2015-01-13 10:59:25 UTC
Created attachment 979562 [details]
engine log amd db dump

Comment 6 Elad 2015-01-13 11:11:56 UTC
Created attachment 979568 [details]
vdsm log

The failure in vdsm:


769a74e3-b33a-4c67-a3b5-ed2803ee795d::ERROR::2015-01-13 12:08:35,005::task::866::Storage.TaskManager.Task::(_setError) Task=`769a74e3-b33a-4c67-a3b5-ed2803ee795d`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/storage/task.py", line 334, in run
    return self.cmd(*self.argslist, **self.argsdict)
  File "/usr/share/vdsm/storage/securable.py", line 77, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 1634, in downloadImage
    .download(methodArgs, sdUUID, imgUUID, volUUID)
  File "/usr/share/vdsm/storage/image.py", line 1254, in download
    imageSharing.download(vol.getVolumePath(), methodArgs)
  File "/usr/share/vdsm/storage/imageSharing.py", line 168, in download
    downloadImageImpl(dstImgPath, methodArgs)
  File "/usr/share/vdsm/storage/imageSharing.py", line 69, in httpDownloadImage
    methodArgs.get("headers", {}))
  File "/usr/share/vdsm/storage/curlImgWrap.py", line 71, in download
    raise CurlError(rc, out, err)
CurlError: ecode=1, stdout=[], stderr=["fork_exec('/usr/bin/curl', 'curl', '-q', '--silent', '--fail', '--show-error', '-H', 'X-Auth-Token: 8930cf80cf334eb88133088ba6e1ed92', 'http://10.35.160.139:9292/v1/images/0aef12f3-ef2f-4b1e-930e-caea9b388da6')", "fork_exec('/bin/dd', 'dd', 'bs=2M', 'of=/rhev/data-center/00000002-0002-0002-0002-0000000003e1/4b9b6b0a-31c7-4ed4-928b-771ef29e8c60/images/2bf52586-a3f1-4e33-9037-b121574aaab3/122c26a9-7daf-4897-9105-0a8714b95c3c')", "dd: writing `/rhev/data-center/00000002-0002-0002-0002-0000000003e1/4b9b6b0a-31c7-4ed4-928b-771ef29e8c60/images/2bf52586-a3f1-4e33-9037-b121574aaab3/122c26a9-7daf-4897-9105-0a8714b95c3c': No space left on device", '28+605162 records in', '28+605161 records out', '7335698432 bytes (7.3 GB) copied, 104.057 s, 70.5 MB/s', 'curl: (23) Failed writing body (2560 != 15928)'], message=None

Comment 7 Yaniv Lavi 2015-01-19 13:03:37 UTC
Doesn't seem urgent, will be fixed in 3.6 only, unless a customer will want this for 3.5.

Comment 8 Vered Volansky 2015-01-28 10:56:24 UTC
Elad, which glance image did you use?
The heaviest is 1GB.
Please state all steps to reproduce, since we're out of synch on this.

Comment 9 Elad 2015-01-28 11:40:27 UTC
I didn't use the ovirt-glance public repository. I used one of RHOS QE group's setups.
The image size I used was 7G  (imageSizeInBytes = 7516192768).

Comment 10 Elad 2015-04-26 10:12:31 UTC
An attempt to import an image from Glance repository to a storage domain that does not have enough free space is block on CanDoAction

Verified using ovirt 3.6.0-1
ovirt-engine-3.6.0-0.0.master.20150412172306.git55ba764.el6.noarch

Comment 11 Allon Mureinik 2016-03-10 10:46:26 UTC
RHEV 3.6.0 has been released, setting status to CLOSED CURRENTRELEASE

Comment 12 Allon Mureinik 2016-03-10 10:49:30 UTC
RHEV 3.6.0 has been released, setting status to CLOSED CURRENTRELEASE

Comment 13 Allon Mureinik 2016-03-10 12:06:01 UTC
RHEV 3.6.0 has been released, setting status to CLOSED CURRENTRELEASE


Note You need to log in before you can comment on or make changes to this bug.